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ABSTRACT 



; This report .reviews theoretical and empirical studies of' 
decisionmaking. The purpose of the review was to identify results 
i.^ that .would be app-licable to the problem of training decision makers. 




\ .Dec^slon-^^Tlaking is conceptualized here as a type of problem / 
^solving, and the review is organized in term^of the following , 
component tasks: information gath'ering, data evaluation, problem 
structuring, hypothesis generation, hypothesis evaluation, pref- 
erence specification, action selection, and decision evalilation. . 
Implications of research findings., for training arlsdiscussed in 
the context of descriptions of each of these^ tasks. 
/ . . • ^ ■ .... 

general conclusion drawn from the study is that -decision 
making-is pjrobably not sufficiently wel]> understood to permit the 
design of .an effective general-purfJose training system for decision 
makers. ■ Systems^ and programs could be developed, however, to 
facilitate training with.resppct to sp'ecil'ic decision-making skills .- 
The development of more generally applicable training techniques 
or systems shouM proceed in an evolutionary fashion. 

*' • " ■ 

Training" is one way tQ improve decision-making performance; 
-another is to provide the decision maker with aids -for various ' ' 
aspects of his task. .Because training and the provision of decision 
aids are viewed as complementary approaches to the "same problem, 
the report ends with a discussion of several decigion-aiding tech- 
niques that are in one or another" stage of study or development. 
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FOREWORD 



The Human Factors Laboratory ofi the Naval Training Equipment/ 
Center has been involved in decision-making research with the . 
objective of developing an approach to decision-making trainii^ 
which yill improve the decision-making and tactical performance 
capabilities* of Navy commanders* This report is the .result of an 
^ analytical review of decision-making research which was perf ormqc^*, 
to identify information pertinent to the training of debisiipn- 
making skills. - - * * - \ 

The outcome. of %his effort corroborated an ijnpression that 
very little of the great amount of decision -making ifesearch . has 
directly addressed the problem of training in ^decision making. 
The review has Identified* implications ft?r ,the training of decision 
makers and areas for research which could provide insight for the 
development of effective training pro.cedures' and programs. 
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WILLIAM P. LANE 
* Acquisition Director 
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SECTION I ' , " ; , . 

N * INTRODUCTION . 



*Much has been written about the importance of decision making 
for industry, for government, for the military and for rational—or; 
at least reasonable— people in general, Moreover, a. great deal of 
reseaJc}! has been conducted , on decision-making behavior. In spite 
of the'se" facts— or perhaps because of them— there ^is not general 
agreement concerning what decision making- is, how it should be done 
how it i£ done, how to tell'. whether it is -done well or poorly, and 
how to train people to ^o it better. . ' ^ ■ • 

■ / ' 

■ ' The^ term "decision making" has' been applied to a very broad 
range of behaviors.- The detection o'f weak sensory stimuli has been 
viewed , in part, as a decision process (Green & Swets, 1966) , as 
has perceptioi) by hurftans more, general Ly (Bruner, 1957). Pattern 
classification by max::hines (Sebestyfl-/ 1962), the retrieval of 
information from memory (Egan, 1958), the performance of skilled 
tasks such as automobile driving (Algea, 196'4) and airplane pilot- 
ing (Sza'fran, 1970), the production' of sp,eech (Rochi^ster Gill, 
1973), educational counseling (Stewart & Winborn, '19j73) , the pur- 
•chafeing of industrial products (Reingen, 1973), the evaluation of 
theVperformance of salesmen -(Sheridan, & Carlson, 1972) , -and the 
con<Kicting of a laboratory experiment ^(Edw&rds,. 1956)^ are also 
representative .of^ the types of processes that have been di^cusged 
under .the rubric of decision making. Probably when the term is used 
in industrial, governmental and niilitary contexts, however, what 
the user has in mind'^Ls something close to, what Sclirenk (1969). . 
describes' as "situations -characterised by fairly wellrdefined 
obDectives"; significant action alternatives, relatively high ^ 
stakes, inconclusive informatioh and limited- time for decision's, 
(p.' 544). We hasten to add that to.limit one's attention to y 
situations that -Have all of these characteristics would preclude 
consideration of the large majority of ^^j^per imental tnv^sti gabions 
of dec is ion, making, -.in particular, in very few labotatoi^y studies 
of decision making have the stakes been .high; and one may question 
in many— if not most— cases the ^i^ifica^ce of the action alter- 
natives to the experimental sub.je^Fs. It does not^necessarily 
follow that the results -of lal?oratory studies have no relevance to 
real-life decision 'making, of course. -The degree to which one is 
•willing to ejxtrapolate from the one situation, to the other depends 
on the extent to which one subsc/ribes to the view that simple and 
inconsequential decision -probleriis are" solved— at least m principle 
'—in the same ways as are "those that are complex and consequential. 

as' Schrenk(19^9) has pointed put, there are three ways to 
imptove the performance of the human decision element in a system: 
'(1) selection (insure that decisions are made only by individuals 
who are competent to make them) , (2) training (attempt to impxc^ve 
the decision-related skills M people in decision-making positions) 
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and (3) decisian aiding^ (provide decision maker-s -with procedural 
and technical aids to '^compensate f or' thei'r (DWn limitations). To 
the extent thcvt |>erformance of a decision-making ^ system is of in- 
terest, as opposed to that of a human being, another possibility 
that deserves considjeration is tha£ of automation (have machines 
, perform^ those 'decision tasks that they can perform better than " 
' people) . 1 \ , . ^ ' ' ^ 

The niunber*o^ ta^^ks that are now performed by machines that 
t ware once thought to require human skills is growing and will 
continue to do so. Many tasks that involve decision making by 
some definition should^ be-- indeed, many have been — automated. 
Thejce' is little justif icfation for was^cing a good, human brain to 
make what Soelberg (1967) calls "programmed decisions/' decisions 
that ar^ made with sufficient frequenc^^r and unde'r sufficiently 
specifiable conditions to permit the det-ailed description of pro- 
cedures for making triem. Thermostats, g6vexpprs^ regulators, 
stabilizers, computer algorithms, and such things, are the pre- 
ferred "depision makers", for these typ^s of situations. The 
situations with which we are primarily concerned are not of this 
straightforward programmed type. 'They are situations that are 
nove'l, unstructured or unplanned fo^, or .they involve human pref^ 
erenpes that are^ not e^asily Specified, or pptential action con- 
sequences that, are not known with certainty. Clearly, these 
types of situations a^* the more interesting objects of study, 
and are probably more repr,esentative of what people view as bona 
fide decision making. 

, ^ It is impbrtant to recognize that the object-ives of much 
decision-making research are to make navel situations less novel 
by providing prototypes ±n terms of which the novel situations 
can be perceived, to facilitate the imposition of structure on 
situations when apparent structure is lacking, and to provide 
techniques for decreasing the probability of surprises and for 

, -coping with unplanned- for situations as though they had been 
anticipated all along. But the reader who might think that such 
objectives could, if realized, take the charm out of decision 
marking may ^rest easy'; There seems little dang'er of success to 
the point of reducing ail decision making to an algorithmic 

♦process In the near future. Indeed, there are some aspects of 
decision making that men may never feel cbmfortable turning over 
to machines. Hence, the needs for selection, training and de>- 
cision aiding are still real, and are likely to continue to be 
for ^spme time to come. Moreover, as more and more of the pro- 
cedurizable tasks that were once performed by men do becor^e auto- 
mated, the tasks that are- left to be performed by men — or perhaps 
by men and macl?ines in collaboration — take on added interest and 
significance by virtue of their very :5^sistanGe to automation. 
Should not those tasks which seem to require the attention of 
human brains be the tasks that hold a unique fascination for us 
as human beings? 
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The geheral question that motivates this study is the question 
of whether individuals can^ be trained to be effective decision 
makers in unprogrammed situations • And if the answer to that 
question appears to be yes^ the next question that presents itself 
is that of how that trainirfg can be accomplished most'*ef f ectively. 
Immediately/ one is Xe^ to^ more specific questions. Does it make 
sense jbo ""think of decision, mailing as a skill, or as a collection 
of skills, that 'c Sin be deyelop'fed ih a sufficiently general^ way that 
they can be applied in* a^ variety of specific contexts? What is it 
that the decision maker needs to be taught? Cpncepts? Facts? 
Principles? Attitudes? Procedures? Heuristics? 

The literature on decision-making research is volumious, but 
despite numerous references to the importance of the training of 
decision makers (e.g., Edwards, 1962; Evans & Cody, 1969; J^lfming, 
197-0- Hammell & Mara, 1970; Kanarick, 1969; Kepner & Tregoe, 1965; 
Scalzi,'l970; Sidorsky & Simpneau; 1970), the number of studies 
that 'have explicitly addressed the iguestion of exactly what should 
be? taught and. how the teaching can best be accora|)lished is remark- 
ably small. The central interest in the area continues to be 
with parameterization of the decision maker and his environmgnl;^^^^ 
and with generation of specific aids to -the decision process. ^ 

This review is not limited, therefore, to studies that have 
focused specifically oa the issue of, decision training. We have 
attempted instead to look at a rather broad cross section of the - 
■general decision-making research literature with a view to finding, 
wherever we coul'd, implications for Ibhe training of^decision makers 
and clues concerning ^ what further research might lead to more 
effective. training procedures or programs. 

\- • ■ 
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^ • SECTION . ' ' 

SOME COMMENTS ON Ej^CISION THEORY 

One can^'distinguish two rather different^ approaches that 
have been taken to the study of decision malcing, *One is analyti- 
cal; the^ other is "^Dasically empirical • A common goal of both 
.^.pproaqhes, however, is .the development' of forma^l mod.els of 
decision processes. Xn the first' case\ one tries to analyze^ 
decision situations — often hypothetical situations^ — abstracting^ 
from them their commori elCTtients, * One then attempts to produce * 
a^ model of the decision^making process, using the Constructs that 
have been identified in ihe process of an^ysis. In tl^fe empirical 
approach, one begins by observing individuals making decisions in 
realrlife situations-, and attempts, on the basis of these obser- 
vations, to develop parsimonious descriptions of decision-making 
behavior, • ' 

Each a^Jproach* has its strengths andVits weaknesses. - The 
mod?],s generated by analysis are likely to be more abstract . than 
thbse developed through observation. AS* a consequence, they' are 
typically more generals Howe^^er, there may be considerabre dif- 
ficulty in applying such models in specific cases. This is true 
because real-life decision situations frequently are not easily, 
describable in terms that an application of a model would r.equire. , 
In cpntrast, a model of a decision-making process that is developed 
by observing decision makers in action is likely to be applicable, 
at l^ast to situations highly similar to that frorft which the model 
is derived. Such models may. lack generality, however^ and prove 
to be inapplicable outside the cont^ext 'in which they are developed. 

2.1- ' Prescriptive versus Descriptive Models 

A prescriptive model indicates what one should do in a given/ 
decision situation? a descriptive model is intended to describe 
what oive actually does . Typically, prescriptive models are the 
outcomes of analytical approaches to the study of decision- making, 
whereas empirical approaches^ generally lead to descriptive models^ 
In theory at least, a prescriptive model may be used either as a 
guide for decision makea^s or as a standard against 'which to assess 
the extent to which^-^lecision-making performance approaches ppti- 
mality. Descriptive models differ from prescriptive mbdels inso- 
far as human decision makers perform in a less^^than optimal fash- 
ions. Were a decision maker to behave in an optimal fashion, a 
'description of his behavior would, constfitute a presci^ipt^ve model. 
Comparisons between .prescriptive and descriptive models can be 
instructive in suggesting the reasc3tns why human behavior is some- 
times not optimal. 

Prescriptive models are generally associated with economists 
and mathematical statisticians. Among the developers and expositor j 
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Of prescriptive decision tlleory are. Bernoulli (17^8), Neyman and 
Pearson (1933), Samuelson (1947) , vonNeumann and Morgenstern 

(1947), Wald (1947, 1950) Good (1952), Blackwell and Girshick . 

(1954), Savage (1954)^ Luce and Raifi'fa (1957), and Schlaifer (1959^ 
Such models typically postulate-^an -''economic , " or at least a 
"rational," man who beha-ves in a way that is 'entirely consistent 
with his decision objectives and who does not have ?ome of the 
limitations of 'real people.. . ' 

Descriptive models wera introduced primarily by psychologists 
ana other students q.£ human behavior, notably Edwards (1954, 1961)- 
Petersdn, Birdsall, and Fox J1954).; Thrall, Coombs, 'and Davis 
(1954); Simon (1954, 1955); Tanner (1956); DavidSpn, Suppes, and 
Sieg^l (1957); Festinger (1957); Luce (1959); siegel .'(1959) ; 
Rapopc^rt (1950); Estes (1961); and Edwards, Lindman, and Sava'ge 
(1963). The' objective in this case has been to' discover by 
experiment ^nd observation how human beings, ^iven their limita- 
tions, perfonri in decision-making situations. It is important 
to np^e that descriptive models have been viewed' as descriptive 
only the behavior of the decision maker, and not necessarily - 
of the ^thinking that leads to that behavior, j'or example, the 
finding \that an individual's choic.e between two gambled can be 
predicted on the basis of which has the ^ost favorable "expected, 
outcome" is not tak^n as evidence that in making the choice the 
.individual actually goes through the process of calculating 
expected values a^d picking the alternative* with the largest one 
(Edwards, 1955; Ellsberg, -1961) . ' 

— \The two lines of development — prescri|)tive' and descrip^:ive . 
models^— -have not proceeded .independen-cly of each other. Several 
of the investigators mentioned above have made significant con- 
tributions of both prescriptive and descriptive types. Moreover, 
one approach that has tJeen taken to the- study qf human* limitations 
IS that of attempting to modify prescriptive models st) that they 
are* in fact more descriptive.. Typically, what this 'involves is 
t^e imposition of constraints on the model that represent specific 
limitations of. the human. For example, a prescriptive model that, 
assumes an infallible memory^^ unlimited capacity is unlikely 
to be very descriptive of hi/fffan behavior; tp modify such a model 
forf the purpose of increasing its descriptiveness would necessi- 
ta^tte at least the addition .of some constraints that represent" ' 
such factors^jas a lamitation on memory capacity and degradation ' 
of stored i-nforihailon over time. 

' . ' / • -< ' 

The distinction between prescripftive and descriptive models 
is sometimes blurred in the literature and one cannot always be 
sure in which way a proponent of *a model intends for it to be 
taken. On the other hand, many writers have observed that the 
models deriving from theories of economics do, in fact, fail to 
describe behaVior,*or at least to d6 so very accurately. Miller 
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and Starr (1967) poiilt out, for example, that in the economist/ s 
view of decision .making, the ^objective of the decision ftiaker is to 
maximize the "utility" that he can achieve within the limitations 
^of his resources* JThey ,note, however', that the assumption that 
individuals ^ act so to maxirrdze utility has been cfthlienged 
by many investigators of decision making. If rationality is 
defined in terms of the extent to which behavior is appropriate 
to the, maximization of utility, they mote, then when people do , 
not maximize utility, they, by definition, are acting irrationally, r 
Miller and Starr list several factors that have been suggested as 
' possible reasons for i:he f ail^ure of 'decision makers to behave in 
an optimal way: "the inability of the individual to duplicate the 
rather* recondite' mathematics which economists have used to solve 
the problem of maximization of utility; the existence of ot^her 
values v/hich, though npt readily quantifiable, do cause divergences 
from the maximization of utility in the marketplace; the effect of 
habit; the influence of social eiftulation; the .^effect ot social, 
institutions" (p. 25). j 

^While interest in prescriptive models stems at least in 
part from the 'assumption that they can provide guidance for 
decision makers in real-life situations, their application often 
proves to be less than straightforward. Haythorn (1961) notes 
the difficulty that operations analysts and operations researchers 
often encounter in trying to analyze decision situations in .com- 
plex organizations to the point that pre scrip ti'^i^ models can be 
applied. He ascribes the difficulty tb sever^ factors: "First 
,is the fact that organizations are constructed by men with some^^ 
/purposes in mind, although these are not usually stated very 
explicitly. Analytic solutions must assume th*at*the decision 
maker is rational, that the parameters relevant to the decision 
are quantifiable, and th^t the information necessary to make an 
optimum decision is ava,ilable. A careful look at the view of 
the world held by critical decision makers reveals that they are 
by no means completely rational? that some of their objectives 
are not easily quantifiable, and perhaps even incompatible with 
other objectives; tnat they do not have ^11 of the information 
needed in many cases; and that frequently the informatioti they 
have is inaccurate" (p. 23). 



Schrenk (19&9) has argued that progress on the development 
of techniques for aiding de^cision makers will- be impeded until 
a model of "optimum" decisio^i processes that makes realistic 
assumptions about human capabilities is forthcoming. Such a 
model, Schrenk suggests, should reflect the behavior of "reasofiing 
man," ^ concept that H^.distinguishes from the rational man of 
economic decision theory "The idea is not to specify an 'ideal' 
decision procedure which v^^ll produce perfect chqices in abstract 
or laboratory situations, but rather to develop a process that 
will yield better decisions in real situations" (p. .548). Schrenk 
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se^s foilr purposes that such a model might serve: (1) it could # 
preside* a framework for tne classification and integration of the 
• resiilts of decision-making research; (^) it co^U-d provide guidance 
f6r fuif'ther research^ (3) it. could help systeS/designers €o 
'structure decision tasks and to allocate decrasion functions to 
men ^nd machiiifes; and (4) it could help guide the development of 
decision-aiding concepts. . ' . . 

'2,2 Horthy PrbbaMlity and Expectation ^ 

* Sometimes a decision m^^ker has the tfsk of choosing one from 
among several alternative courses of action, knowing what the 
effect of any choice would be. (This situation, which is referred 
to* as decision making under cer.tainty, is di*&cuss'ed in Section IX,) 
Often f however, one must make a choice when tife consequ^jtaces of 
' that choice cannot be anticipated with certaii^y. In the latter 
situation, the decision maker is said to be maJfting a decision • . 
"under " risk. " The most common/ way of dealing ^th risky decisions 
quantitatively has been with models that make use of the concept 
of mathematical expectation.' - \ 

* s 

' The "expectation" associated with a choice is calculated h^' 
obtaining the product *(^<*som§ measure of worth^ of* each outcome^ 

l^and a measure of the probability of thcit outcome, and summing over 
•all outcomes that could result from the, choice of interest. It, 
has sometigies be^n assumed that the decision maker attempts to 

' make a choice t*hat maximizes his "expected" gain. More precisely, 
it is assumed that the decision maker behaves as though he calcu- 

. lated for each action alternative, the sum of the products of the 
worths and probabilities of the possible outcomes ^ associated with 
that alternative, and picked i:he alternatij^ for which this sum ' 
was gr.eatest. The "as though" in the preceding statement is 
important • No one contends that decision makers, as a rule, ^really 

• perform the arithmetic necessar^ to compute expectation;' it is 
only suggested that choices are made as though they wer.e based on 
such calculations. : 

Each t)f the factors in the expectation equatidn-^-^wortli and 
probability — can be treated as either an objective a subjective^ 
variable. The four possible combinations of objective and sub-. / 
jebtive indicants of worth combined with objective and subjective 
measures of probability define four classes of -expectation models * 
tha\t have been studied. Table 1 gives expressions, in the nota- 
tioh used by Cooi^ibs, Bezembinder, and Goode (1967), for expecta- 
tions representing each of these models.^ Much Of /-the research on 
decision making under risk has been concerned wit nv determining 
which -of these models is most descriptive of human behavior, and 
with developinq techniQues for measuring subjective worths. and 
probabilities. ^ ' < 
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TABLE 1. FOUR BASIC 4TYPES OF EXPECTATIQN • MODELS . 
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an objective probability 
a subj ec tive probalJility 

an objective measure of value (e.g. amount of money) 
a subjective measure of worth (or utility) 
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^ The first of the models listed in Table 1, the Expected Value 
models is t^e least complex conceptually, and the most easily ap- 
plied, inasmuch as* both of Uts parameterrs are objectively defined. 
Although -bhis model has some appeal as a prescriptive model, it 
has proved jiot to be generally descriptive of' how real decision 
makers behave (^at, for example. Coombs, Dawes, s Tver sky, 1970; 
Edwards, 1961; '/Lichtenstein & Slovic, 1971; Lichtenstein, Slovic, 
& Zink/ 1969) . • 4 > ^ ' . 

The inadequacy of the Expected Value model' as a descriptive 
ipodel' is. clearly illustrated' by the well-known St. Petersburg 
paradox. Suppose one were offered an opportunity to purchase the 
following gamble. A fair coin is to be tossed until it comes up 

. tails, at which time the'cpin tossing is terminated anc? the winnings 
are collected. If the coin confes up heads on the first toss, the > 
purchaser will receive $2.00; if it comes up heads on both the 
first and second toss, he will receive $6.00 (or $2.00 for the 
first toss and $4. Oft for the second). . More generally., if 'it comes 
up heads'fof n consecutive tosses^■ he will receive $2.00 for the 

^ first, toss, $4. 0(^ for the second, $8.00 fpr the " third . and $2^ 
for the kth, for a total of V 



n 
Z 
k=l 



» S 2.^ ddllars, 



Since, by definition, the successive tosses' are independent, the 
expected value of this gamble in* dollars is .given by 

EV = I • 2 + i • 4 + . . . + 2" +*. . . = 1 ^ 1 + 1 + . . ; 

^ which is to say, it is infinite. if one were attempting to maximize 
expected value, therefore, one should be willing to pay a large 
amount of money indeed to play this game. It would be surprising, 
however, if mariy people could be found who would be willing, to 
risk' their life savings, say, which would be small by comparison 
with the expected gain, to purchase this gamble. In general, It 
is clear that the attractiveness of a gamble depends not only on 
the expected value of the outcome but on such factors as the amount 
that one couldv possibly lose, and the. nature of the distribution 
of probabilities over. the possible outcomes. In the gamble de- 
scribed above, lOr example, the probability is '. 5* that the. purchaser 
will win nothing, and .75 that he will win "at most $'2.Qb. 

1 

iB- spite of thd inadequacy of th^ Expected Value model^sya 
genera :^ly vaiid description of behavidr. it should be noted tfTat " 
the model does a creditably good job of describing behavior, at 
-least gros9ly, in many decision situations. Even in the case of 
gambling behavior, it does not "invariably faAl; '"about 88% of the 
job" of .explaining the behavior of the\ Las Vegas gamblers studied 
by Edwards, ^for example, could jDe done on the basis'of a knowledge 
of the expected value of each bet (Rapoport & Wallsten, 1972) 

V: 
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Implicit to the Expected Value model is^tl^e assumption thaf^ 
the monetary value of a decision outcome represents its real worth 
to the decision maker, and tl^at ,thi^ worth is the same for all 
individuals ♦ Recognition that such an assumption is undoubtedly 
'false led to the formulation of the Expected Utility mod^l in 
wl^ieh, monetary value is replied by a measure of the "utility" of ^ 
anlSutcbme for the particular decision maker involved. According 
to this formulation the.^ame Secigion outcome may appeal to .dif- 
ferent individuals to different degrees, ahd, consequently, prefer- 
ences, among decision alternatives wjlth uncertain outcomes may / 
differ from one decision maker to anotlier. The Expected Utility 
model was first proposed by Bernoulli ^1738) and aiven its modern 
axiomatic form by von Neumann- ^nd Morgei^tern (1947) / 

Given that the worth factor m the expectation equation is 
defined as a subjective variable, the question arises concerning 
how probability should Abe cjefined. Although a review oi the con- ^ 
trover sy would take us H:oo far, afield', it should be noted that the 
question of what the concept of probability "really means" has 
been the subject of 'endless philosophical debate ♦ It is sufficient, 
for our purposes to recognize that statements of the type "the 
probability of the occurrence of event X is equal to Y" have, been 
used in a variety pf ways. Such ^ statement is sometimes used to 
refer to the relative frequency with which X has been^ observed 
ove^ the course of many similar situations. Or it .can have refer- 
ence to^ a ratio in which the numerator represents the~ tqtial number 
df'ways in which the outcome of an hypothetical experimenis^^can 
satisfy sdmfe criterion and the denominator represents ^the total • 
number of 'different outcomes (as wheji one says the proba- 
bility of rolling a 2 or^ less on a fair die is 2/6). y\. Sometimes 
a probgibility statement* is useff to refer to the strength of one's 
confidence, or the degree of one's belief, that an event X, as 
opposed to the*^other events that are considered possibilities, 
will occur ♦ It is this connotation that we here refer to a's. 
"subjective probability." y/^ 

In some Situations it ^akes little if any praotical ^difference 
which of these connotations one gives to the conc,ept of probability. 
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*Related to this usagg of the term is the so-called "Principle of 
Insufficient Reason, " which cjirects the decision maker to consider 
all possible outcomes tp be of equal likelihood in the absence of 
information which indicates such a consideration to be inappro- 
priate. See Rapoport (1964) for an 'interesting discussion of the 
limitations of this prescription in defense Otf an assertion that 
the six faces of a die are equally likely when one has no reason 
to assert otherwise. ^ ' ' * 
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inasmj)ich as they will all yield trhe ^ame, num£)ers. Most people 
woula perhaps agree, for example, t*hat the probability of tossing • 
head^ on a fair cpin 'is . 5, * irrespective of their philosophlbal' ^ 
position concerning how probability should be jJefined. Many 
"probabilistic" situations of interest to ' investigators of de'ci^ 
sion making do not easily admit of an analysis in terms of rela-^ 
tive frequencies, dr even of theoretickl ratios, however, and it 
is perhaps for this reason that many decision theorists subscribe 
to the notion thaf probability is best defined" in terms of** degree 
of belief. Rapoport (1964) defends this positionjthe following way. 
"We are told that decisions involving the probability of the out- 
break of a nuclear warfare based on 'calculated risks,* by which 
term thoi^e who recommend or make deci'&ions must imply calculationsf 
involving probabp-lities. Since tpe'prpbability of an event such 
as the outbreak of a nuclear war can have nothing to do with the 
f requency^jOLf^ such events Tsinc^ at this writing none has occurred, 
and,'in/ali likelihoo'd-, no more than very few can occur)*, either' 
thg,-4>ilrase **the :probability of 'a nuclear war* has no meaning at , 
all, in* which case the notion of the 'calcu^lated risk' is only 
eyewash, or else 'probability* has another meaning, having nothing, 
whatsoever to do with frequency" (p. 25). ^ ' ; : 

The argument that* probability often 'cannot be defined mean- 
ingfully in terms of relative frequencies or ratios* is a strong 
one for» resorting ^o a definition in terms of subjective uncer-^ 
tainty. Even when an objective definition is easy to come *by, 
however, one may question .whether it should be ysed by any ^theory 
that purports to be d^criptive of the behavior of real decision 
makers. It it the decision maker ^s own expectation that i^ pre- 
sumably important in determining his behavior and his expectation 
mu^t be calculated in t^rms of the, probabilities as he perceives 
th^. Moreover, it is required of a rational man that Bis behavior 
be' consistent with the information at his disposal, but not that 
he have perfectly accurate iaformaj:ion\ Thus, two*decision makers 
could behave optimally^but quite differently, in the same situa- 
tion if their perceptions ^'f the situation diffei^ed, a fact that 
is easy to accommodate when^robability is d^fip.ned as degree of 
belief but nbt when it is defined strictly in terms of the ob- 
jective details of the situation. 

In the foregoing discussion of Expected Valup and Expected 
, Utility models, it was tacitly assumed that the probability factor 
in thQ. expectation equation was objectively defined. As suggested 
by Table 1/ two additional types of expectation models might be 
realized by combining subjective probabilities with both objective 
and subjective measures of worth. 'The resulting models might be 
referred to, respectively, as Subjectively Expected Value and Sub- 
jectively Expected Utility models Although both of these types of 
models have been considered, the latter is by far the more widely ac 
cefJted .and, used. This modSl has been presented by Savage (1954) and 
by Edwards (1955). Among the four models listed in T^able 1 which 



11 

24 



I NAVTRAEQUtPCEN 73-C~0128-l ' ' 



have been referred as single-stage algebraic decision models- 
it has t^ceived theSgreatest amount of empirical support, and 
at the-.moment, ranj^.as the most influential (Rapoport & Wallsten, 

• 1972) . . * ' * ' ' 

( " , . 

Y Savage^s (1954) formulatior; of decision theory ident'i^fies a 
nuinbet of "seemingly agreeable" (Tversky, 19^9) rules that 
should be satis.fied bsfojfe it is appj^opriate to assign a single 
f ixe^a number denoting- worth to each possible decision outcome and 
a single fixed number denoting judged^^'likelihood ■ of occurrence 
and then to 'select maximum products. These rules (sefe Beck^ & 
McClintock, 1967) ■ ^e\as follows: ^* 

RujLg 1; T-ransitivity . If, in choice situation, the de- ' 
cision/inaker prefers Outcome A to Outcpme B and Outcome B to 
Outcome he should prefer. Outcome A to' Outcome C. . . 

Rule 2; Comparability . The decision maker "should b^ willing 
to compare two possible outcomes arid decide either that he prefers 
one to the other or that he has no 'preference between them. 

Rule 3; Dominance . ^ If the' decision maker determines that, ■ 
under ev ery possible donditiori a.* choice of 'on^ of ^is alternative 

• aqtions res.ults* in an outcome at least, ag defirahle as that which 
would result from the, choice of a second alternative action, and 
results in a more defeirable outcggme ' und^r ^Ueast one possible 
condition than would .the second ab^ioh, the secohd action should 
not be preferred to the first.' ^ ' " ■ " . / ' ^ 

.• Rule 4? Irrelevance of "non affected outcomes . If the de- 
cision maker determines, that, fop a particular state of the world, 
Jwo or more of the actions open to hin) result in the same outcome,, 
his preferences among such actions should not be affected by the 
•outcome associated with that state, 

' . ■ • . -u . ■ 

Rlile 5h- Indepertdenoe of beliefs and rewards . The decision 
maker s statement concerning the likelihood of occurrence of .a 
given outcome should , not be affected by wha^ he hopes will occur. 

^ome of these rules seem to be honored'^as much in the breach 
as in t-he 5bservance^{see, for example, MacCrimmon, 1968). Vio- 
lations of Rule 1. are of majof significance. This is so because 
tTie a.ssUmption of transitivity of preferences is a necessary 
requirement 'for the' cons€ial6t;ion 6f a consistent ordinal utility 
function. (For a discussion of the problem of generating utility 
functions from preference j-udgments, see- Roberts, 1970.) Tversky 
(1969) refers to transitivj.-ty, as "the cornerstone" o£' decision 
theory and points out that it underlies measurement models of 
sensation and vvalue Ss well.. He also notes that decision' makers 
of.ten do violate the transitivity fule in specific situations. 

' - / • * 
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^ ' * ' ^ 

Another rule which seems difficult to satisfy is that .re- 
quiring independence of beliefs, and rewards (Rtxle 5) . MacCrimmon 
( 19*68) has found a strong dependency between an individual's * 
estimates of - the Likelihoods of events and his "tastes" — th,§ 
worths he assigns to those events. 'As might be anticipated/ such 
an associati,on may pose difficult analytic problems, since, for^ 
a given set of choices, one cannot assume that a distribution of 
(stated) preferences arises simply ojat of differences associated 
with but one of 'the two parameters .in the expectation equation. 
In principle, this problem is similar to the so-called "conjoint 
measurement/ problem" which has rec^iv^d major attention in the 
context of Subjectively fixpected Utility theojry. 

"^variant of the dominance principle (Rule 3) has been stated 
by MacCrimmon so as to apply .to the problem of comparing a.lter- 
natives that ""differ with respect to several -attributes .when 
preferences can be stated with respect to single attributes 
individually: . "W^en ^comparing all alternatives, if some alter- 
native has higher Attribute, values for all attributes, we say 
that this alternative 'dominates' the oth^ers. We can weaTcen this 
•not:j*on somewhat and say that if one alternative is at least as 
good as the other alternatives on all attributes, and is actO^lly 
better on at least one of them,^ then th.is can s-^ill be considered 
the dominanf^ternative. Conversely, if one alternative ih 
worse than some X)ther alternative for at least one attribute, and 
is no better than equivalent ^or \ll other attributes, then we 
can say the former alternatiT>e is dominated bx the latter'' (p. 18) 
Some writers have noted that the dominance criterion is inconsis- 
tent with the maitimin criterion of game theory (M^rschak, 1950) 
Luce & Raiffa, 1957). Ellsberg (1961) has discussed additional 
probl'ems with this rule. 

Somfe other assuinptions that have usually been considered - 
necessary to the use of expectation models are the following: .(1) 
that the act of gambling has no utility its'elf ; ^ (2) that the sub- 
jective probabilities associated witli the alternative decision 
outcomes sum to unity; (3) .that preferences are independent ,ctf the 
method by wjiich they are 'measured. It has not been possibleTTo 
demonstrate that the first two of these .assumptions are simulta- 
neously valid. Moreover, SloVic (1966) and others (Lichtehstem 
& Slovic, 1971; Lindman, 197.0) have shown that 'preferences among 
gambles may indeed depend in part on- the method by which they 
are obtained (e.g., a rating procedure V;ersus a bidding procedure) 
In spite of these limitations, expectation mode^Ls, and in par- 
ticular the Subjectively Expected Utility model, have proven to 
be reasonably predictive of at least certain types of choice 
behavior (CoombS/ Bezembinder, & Goode, 1967). They clearly do 
not, however, tell the. whole story of how to account for human ' 
choice behavior. 
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The demonstration that expectation models such as those de- 
scribed are unable to account for choice behavior' .consistently 
and completely has- led some theorists to seek to modify (which has 
invariably^meant to complicate) the/ models to make tl\em more de- 
scriptive. ^ Other- theorists have simply rejected them out of hand. 
Payne (1973) points out that models ^such as those we have con- 
sidered involve the representation of risky alternatives as proba- 

-bility distributions over sets of decision outcomes, and attribute 
the choice among the decision alternatives to some function of each 
distribution's central tendency. In the hope of developing models 
with greater predictive^ power, some theorists ha,ve looked not only 
to central tendency measures, such as expected or mean values., 
but to variances and higher moments of thes^ distributions as well 
(Backer & MeClintock, 1967) . Still others have made modifications 
J[±at: relax the requirement 'that the decision maker's choice be f 

.invariably dictated by which of his alternatives represents the 
greatest expectation; /'random utility " models have been proposed, 
for example, which assume that the utility of a- given outcome 'is 
a random^ variable and that variations in this variable produce 
variAtioGS in choice (Becker, DeGroot, & Marschak, 1963) . 

Shackle's (1967) asse^^ment of 'expectation models is represen- 
tative of the* opinion^ of thedrists who reject such -models out of 
hand. He argues that the concept of mathematical expectation, 
and, indeed, ^the concept. of probability as well, are irrelevant 
to' the assessment of one-of-a-kind decision situations. Further- 
more, \he contends, most real-life decision situations of interest 
are, to ^those who face them, unique events;, never befote has the 
individual been->called upon to .make exactly the choice that he 
faces And never again will he h^ve to select from among the same - 
set of action alternatives under precisely the same circumstances. 
In such cases. Shackle argues, the decision maker is concerned 
with what can happen as a* result of his choice, hot with what 
would happen if the experiment were repeated a large number of 
time^* "he is concerned with possibility 'and not probability " 
Ip. 40). We should note that the argument implies a relative-, 
frequency connotation of probability, a connotation that not all 
decision* theori^sts accept. 

Miller and Starr (1969) suggest that one can always find a 
way J:o view a decision problem as^.-aT^aximization problem if one 
wants to do so: the quantity>feliat^ the decision maker wishes to 
maximize is the degree of attainment of his objective. But this 
is nQt sff^ry helpful as a definition: indeed, it comes close to 
Jjeing* tautological. Millet and Stai^r^ apparently do not intend 
to assert as an empirical fact that decisioh makers do attempt to \ 
maximize anything. More generally whether 'decision makers attempt 
to find optimum solutions to their decision problems Miller and 
Starr consider to be questionable. Simon (19 55) has taken the ' 
position that they usually do not. According to his "principle 
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of bounded rationality" what they do instead is -to def ine a ^ 
limit:ed set of acceptable/ or "good enough/" decision 'outcomes 
and then select a strategy that they consider to be likely to 
achieve one of these. / 
- * » • ' 

The current status of expectation models among investigators 
of decision making is, reasonably- well summarized by three obser- 
vations. (1) The models that are seriously advocated as descrifJ- 
tive^of human behavior are rather more complex than the straight- 
forward Expected Value model that was origihally proposed. * JThe 
history of the development of expectation models may be fairly 
characterized as a progression from the simple to the more complex: 
objectively defined variables have been replaced with variables 
defined in subjective terms / and the^ number of. .model parameters \ 
has been increased. (2)- Even the most ^complicated models have 
not proven to be total3,y descriptive of behavior and some theorists 
have challenged the validity of the basic assumption of this class 
of models/ na.aely that the decision maker is motivated to maximize 
.an expectation/ no matter how the j factors from which expectation 
xs computed are defined. (3) Their limitations notwithstanding/ 
expectation models — even the least sophisticated*Expected"Value 
model — do a reasonably good job of predicting choice behavior in 
many situations. The challenge i's to come up with models that 
can hand^Le the situations for which these models fail/ as well * 
as those for which they succeed. Meanwhile, when ''the maximization 
of expectation is recognized as the decision objective, then 
expectation models can be used prescriptively to guide the 
decision process. ^ ^ ^ ^ 

2,3 G'Ane Theory ^ . 

The theqry of games was developed to deal with situations 
in which the outcomes of an individual's decisions depend ^not only 
upon his own actions but also upon those of one or more "opponents" 
— decision makers whose objectives conflict to some degree with 
his own. Of specials-Interest is the so-called "zero-sum" situa- 
tion in which the worths of the outcomes to the opponents sum to 
zero; one loses what another wins^ , 'A. commonly prescribed strategy 
for each "player" of a zero-sum game is to make choices in such 
a way as to minimize his maxiirium possible loss/ the so-called 
minimax rule. ' - 

The ass^lmptions.of game theory are open to a number of criti-/ 
cisms. Shackle (19G7)/ for example, characterizes the tFieory 6f 
games / as developed by vonNeumana and Morgenstern/ as "essentially 
a study of the logic of hoV to present as impregnable a front 
as possible to an infallibly wise and rational opponent" (p. 61) . 
The assumption that one is in a conflict and that one's opponent 
is rational and infallibly wise leads directly to the minimax 
doctrine. Shackle questions to what extent this conceptualization 
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can be taken as a reasonable approximation to reality. "is the 
impersonal world of nature or even ^that of business actively con- 
cerned to defeat us? Is the humap opponent reasonably assumed 
to be infallible? Is there no essential and ineradicable uncer- 
tainty in the outcomes of such few big experiments, large in time 
scale in comparison with the human life-span, that any of us has 
time to make? Rather than minimax our losses, is it not more 
^reasonable to fix for them some maximum tolerable numerical size, 
to avoid any aotion-scheme which would bring losses larger tfian 
this within the range of possible or . 'too-possible' outcomes, and 
subject to this, constraint to choose that action-scheme which 
brings within the range of., possible 6r 'sufficiently possible' 
outcomes, as high a positive success as we can find?" (p, 65). 

In a similar vein, BeckW and McClintpck (1967) question 
what they refer to as game .theory's "principle psychological assump- 
tions." They point out that the theory assumes, on the one hand, 
that both decision makers will attempt to maximize their own 
utiiity and, on the other h^nd, will attempt to minimize their 
maximum losses. These assumptions are, inconsistent unless the 
decision makers lock at the game f rom<. each dther's points of 
view — a requirement which Morin (1960) finds unsupportable on 
empirical grounds — and unless the utilities of each decision maker 
are known to the other and sum to zero for each possible outcome. 



Despite its limitations, game theory has provided a valuable 
framework within which to view decision making. in suclv^ields as 
economics, political science, social psychology and mili^tary 
strategy. The theory has been* extended ±o cover non-zero-sum 
^situations, situations permitting pooperation or collaboration 
, among sutfsets of players of multi person games. In addition to 
minimax, other strategies have been identified as either prescrip- 
tively appropriate, or descriptive of behavior, in particular 
situations; 

, ' A ^hort and very readable exjjosition of^ the basic concepts 
of game theory may be found in Edwards (1954) . A comprehensive 
tutorial treatment is provided by Luce and Raiffa (1957). 

2.4 Decision Theory and Training \ 

^ It is a reasonable question to railse whether one may hope to 

be an effective decision maker in a variety of situations without* 
some intellectual appreciation for the decision-making process, 
as it is. represented by theoretical treatments of decision making. 
One would guess that there would be some advantage to being famil- 
iar, at *least with certain of the key concepts that decision 
theorists employ. In practice, this would mean providing would-be 
cAdecision makers with a basic introduction to probability theory 
. a^Sxwell as a v;orking familiarity with notions of rationality, * 
value, utility, mathematical expectation, risk, risk preferences, 
and so on. 

« 
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In fact, one could make the case that failure to provide an 
adequate grounding in theory might deprive the decision maker of 
the sorts of insights that wouJLd lead to pjroductive use of avail- 
able decision-aiding techniques. The demonstration by MacCrimmon 
(19 68) that decision aids developed in quite disparate contexts 
can be effectively brought together in the solution of problems 
involving multi-attribute alternatives, suggests the utility of 
broad acquaintance with basic concepts and principles. 

In reporting one effort »to develop ^ system to assist cor- 
porate decision makers by enabling, them to manipulate parameters 
(entered as distribution functions) on preprogrammed tree irt^dfels, 
Seville, Wagner, and Zanatbs (1970) made some observations that 
are relevant to this point. They noted tha^ the use of subjective 
probability distributions as inputs to models is novel even to 
experienced decision makers, and must be c^refuH^y taught. More 
generally, they concluded that a black-box approach to utilization' 
of'* the system would have been markedly inferior to one in which 
the workings of the system- were explained to the user. 

• ' The teaching of decision theory should, of course, distin- 
guish what is inten(3ed to be prescriptive from what is considered 
descriptive of the behavior of hutnan decision makers. It should 
also clearly identify the limitations of the models that are 
considered. Tutorial treatments of decision ttieory and game 
theory are readily available sources of training material (Edwards, 
1954r Edwards & Tversky, 1967; Howard, 1968; Lee, 1971; Luce 
& Raiffa, 1957; Miller & Starr, 19 67; North, 1968; I^apoport, 1960; 
Schlaifer, 1969) . A comprehensive bibliography of research reports 
has been prepared by Edwards (1969). 

Whether familiarization with theoretical treatments of de- 
cision making will in fact improve decision-making behavior is a 
question for empirical research. Our gu.ess is that the- answer 
will be a qualified yes. ' Such training will be efficacious for 
some people performing certain types of decision tasks but perhaps 
not for all people or all tasks. One objective of training research 
should be to i'dentfSy those conditions under which such training ^ 
would be effective and those under which it woul'd b? a waste of 
time.' 
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SECTION III 

CONCEPTUALIZATIONS OF DECISION SITUATIONS AND TASKS' 

Numerous ways of conceptualizing decision processes have been 
proposed by different investigators'^. Some conceptualizations 
emphasize differences among decision situations/ others focus on 
the tasks that decision makers are recfuired to perform. All of 
them have the same purpose, namely that of simplifying the problem 
of thinking about decision making by identifying a 'few "types," 
each of which 'is representative, in terms of some critical aspects, 
of a variety of specific situa^tions or tasks. We review briefly in 
this section a nxmiber'of p;;oposed simplifying conceptualizations. 
The^re is\n0* attempt to be exhaustive. The inten^t is simply to 
illustrati by raeans of a few examples some of the ways,, in which 
investigator;s of decision making have characterized or categorized 
the object of their study. ' 

. . . / - . 

3.1 Classifications of Decision Situations or Qg ci^sion Types 

3.1.1 Edwards 

Edwards (1967) makes a distinction between static and dynamic 
decision situations. In the former case, a one-time decision is 
required, whereas in the latter, sequences of decisions are made, 
earlier decisions and their outcomes having implications for sub- 
sequent ones. Six types of dynamic decision situations are 
distinguished on the basis of such factors .as whether the environ- 
ment is stationary or nonstationary , whether or not the environment 
is affected hy the decisions that are made, and whether or not the 
information about the environment is affected or controlled by < 
those decisions. Edwards further classifies psychological research 
relating to decision making under four topics: information 
seeking, intuitive statistics, sequential prediction, and Bayesian 
processing. 

3.1.2 Howard 

Howard (1968) characterizes decision situati^Dns in terms of 
three orthogonal dimensions: degree o£ uncertainty, degree of 
complexity (number of relevant variables) , and degree of time ' 
dependence. The various combinations of the extreme values on 
these dimei)sions are taken as representative" of eight prototypical 
, situations, for each of which there is an appropriate set of 

analytidal tools. An example of a deterministic (no uncertainty), 
single variable, static (time-independent) problem would be to 
determine the largest rectangular area that can be enclosed v;ith 
a fixed amount of fencing. , The appropriate mathematical tool would 
be the calculus. Decision problems like assigning customer.^ to 
warehouses or jobs to men would, in Howard's taxonomy, be in the 
category defined as deterministic, complex (many variables), and 
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Static. Matrix algebra and 'linear optimization are appropriate 
mathematioal techniques, 

3.1.3 Sidorsky • ' ' . v. 

Sidorsky and^his, colleagues have proposed a taxonomy of types . 
of decisions encountered in ^tactical military situations (Sidorsky, 
Houseman, & Ferguson, 1964; ^Sidorsky & Simoneau, 1970; Hammell & 
Mara, 1970) . The acronym ACADIA is used as a mnemonic for the six 
types of "situational dema^nds" identifiecf by the taxonomy: 
Acceptance, Change, Anticipation, Designation, Implementation, and 
Adaptation. 

An acceptance-type decision has to do with applying data to 
the acceptance or rejection of a hypothesis concerning some char- 
acteristic of the enemy. Detection, classification and localiza- 
tion are associated operations or objectives. The acceptance- 
decision idea seems to be close to what some other investigators 
have?^ referred to as situation diagnosis. A change- type decision 
invoO^^es the decision, maker in a choice between initiating a new 
tactical operation or continuing the course of action on which he 
is already launched. *- An anticipation-type decision is required ^ 
when a decision maker must predict what the state or intention of 
an enemy force will be sometime in the future. 

^ ^ *• ' 

A designation-type decision involves the choice of one from 
among a set' of possible action alternatives. An implementation- 
type decision has to do, not with the selection of an action 
alternative, but with the determination of the proper time to 
^execute it. An adaptation-type decision is called for when the 
decision maker is faced suddenly with unexpected and perhaps 
potentially disastrous circumstances. 

3.2 Classifications of Decision 'Tasks , - 

3.2.1 Howard ^ ' . / 

Howard conceives of the decision process as Ming composed 
of three I phases: (1) the deterministic phase, (2) the proba- 
bilistic pl}.ase, and (3) the information pha^e. In the deterministib 
phase, the decision analyst identifies the state and decision 
variables and constructs a model of the decision problem. In the 
probabilistic phase, he assigns, probability distributions on the 
state variables. In the information phase, he determines what 
additional information should be gathered to reduce uncertainty 
further. Howard estimates that the first phase represents about 
60% of the total effort of the decisiort maker, while the second 
and third phases represent about 25% and 15%, respectively. 



NAVTRAEQUIPCEN 73^-0128-1 



3.2.2 Adelson 

A taxonomy^ of decision tasks that are carried *out in modern 
military command-and-control systems is proposed by Adelson (1961) ♦ 
Four, types of tasks are distinguished: (1) characterization of the 
state of the world, (2) determination of the available action 
alternatives', (3) outcome prediction and (4) choice rationalization. 
The first task type rpf ers to the need of the decision^ maker to 
characterize the current ^tate of the world in a way .that is / 
relevant to his decision problem. The definition of the variables 
in terms of which the characterization shouM made, and the 
assessment of the relative stability pf the world that is being 
observed are seen as significant problems. The second task type 
acknowledges the need to make explicit the courses of action that 
are open to the decision maker. The difficulty of this,*task may 
depend somewhat on how rapidly the situation is changin*^ and on the 
cost of obtaining information. Outcome prediction refers to the 
j>rocess of attempting to anticipate what the consequences would be 
if specif ic. action alternatives were selected. tJPhe final task type 
involves the need to justify one*s choice of action in terms of the 
objectives of the command-and-coni^ol system. 

3.2.3 Drucker 

Drucker (1967) has identified six steps. that he considers to 
be involved in the, process of making the types of decisions that 
confront business executives: (1) the classification of the problem, 
(2) the definition of the problem, (3) the specifications which the 
answer to the problem must . satisfy , (4) the decision as to what is 
"right (as distinguished from what is acceptable in order to meet 
the boundary conditions), (5) the building into the decision of-^the 
arctioiy to carry it put, knd C6) the feedback v*hich tests the 
validity arid effectiveness of the decision against thfe actual course 
of events. 

3.2.4 Soelberg ; ' . " 

Soelberg^'s (1966) taxonomy, like Drucker 's identifies six 
aspects o£ the decision making process: (1) pa^oblem recognition, 
(2) problem definition, (3) planning, (4) ^search,. (5) , confirmation 
and (6) implementation. 



3.2.5 Hill and Martin ^ 

A model proposed by Hill and^Martin (1971) also recognizes 
six different categories of activities in the decision-making 
process: (1) identification of concern, (2) diagnosis of situation, 
(3) formulation of action alternatives, (4) test of feasibility of 
selected alternatives, (5) adoption of alternative,, and (6) assess- 
ment of consequences of adopted alternative. The model assumes that 
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the decision maker's behavior at each of these steps is influenced 
by what he knows of the theory and practice ^of decision making as 
well as by what he knows about the setting in which the decision 
problem exists. Hill' and Martin identify nineteen skills that 
they consider to be implicil^ in these six generic activity 
categories: 

"1. Asking for and receiving feedback 

2. Assembling the facts (including past experience as it^ bears 
on the decision) 

3. ' Identifying the courses of action available , . . 

4. Identifying forces for and against the alternatives 

5. Ranking and rating alternatives {inpludes.',piitfcing a value 
on applicable risk factors) : 

■ ?. * 

6. Assessing the people-task ratio 

7. Identifying the latest and expected consequences of the 
alternative courses of. action 

8. Determining th^ advantages and disadvantages of each action 
.alternative */ 

'5. Testing the validity and effectiveness of the consequences 
of the decision against the actual course of events to " ^ 
•evaluate the decision maker's judgment and to modify his 
subsequent decision-making behavior 

10. Brainstorming act^ion alternatives 

11. Classifying and defining th^, problem requiring a decision 

12. Analyzing and evaluating stimuli and decisions coming in 
^ from the outside 

Isl Defining the goal at which the decision is directed 

14. Communicating the decision in written or verbal composition 

1*5.. Identifying resources bearing on the making of the decision 

16. Recognizing the need for a debision 

17. Utilizing minor, relatively simple decisions to contribute 
to making the more complex one (includes determining the 
hierarchy of order in which ihino*r decisions will* be dealt 
with and coping with timing as alternatives come into focus 
and seemingly demand attention at the same time) 

18. Obtaining j^nformation 

19. Specifying the boundary conditions the decision must 
* ' satisfy" (p. 4 33) . 
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3. '2. 6 Edwards 

» • 

^ Edwards {1965b) lists the following thirteen steps that must 
be carried out by any Bayesian decision system: 

"1. Recognize the existence of a decision problem 

2. ' Identify available acts a, ^ 



3. Identify relevant i^tates that determine payoff 
for acts . , ' 

• . 4, Identify the value^'^dimensions to be aggregated 
into the payoff matrix 

•5. Judge the value jof eacfh outcome on each dimension 

6> Aggregate ^ value judgments into a composite 
payoff matrix* , " ' ^ 

7* Identify information* sourcefe^ relevant to 
, discrimination among states 
■• 

8*' Collect Mata from information sources" 

*■ 

-9* Filrer data, put.intd standkrd format, 
and display to likelihood estimators 

¥ • 

10, Estimate likelihood ratios (or sonve»other 

quantity indicating the impact of the datum ^ - 

'<on the hypotheses) 

11* Aggregate impact estimates into posterior 

distributions , / • ^ 

12, Decide among acts by using' pjcinciple of 

maximizing expected value ^ ^ 

.13 • Implement t'tje decision" (p. 142, Table 1). • 

Steps 1 through 5 rand 7 and 10, Edwards suggests^ are best per- 
formed by men. Steps 6, 11 and 12 by machines and Steps 8, 9 and 
13 by^ both men and machines. Steps 1 through 7 may be done in 
advance of the decision time; Steps 8 throughJ^13 must be done at 
the time that the decision is to be made. . (See Section VIll for 
a^ discussion of Bayesian information pirocessing. ) ' 
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3. 2*7 Schrenk 

* A conceptualization off^the decision-process that we 
particularly interesting, is one proposed by Schrenk (1969) . ^ The 
motivation for developing this (conceptualization was to prdvide a 
representation of th^ decision-making process that is prescriptive 
in the sense that it can be used 'as a guide for the structuring 
of decision-making tasks of man-machine sy stems ,-^ut which does not 
make unrealistic assumptions about human capabilities. The ^con- 
ceptualisation is viewed by Schrenk as tentative^ and in need of 
further development; howevfer, 6ven as it stan(Ss it provides the 
system designer with a great deal of food for thought concerning 
how to allocate dfecigion functions among men and^ machines 

Three major categories of decision tasks, qr phases' of the 
decision process are distinguished: (1) problem recognition^ 
(2) problem diagnpsis, and (3) action selectioii. Each of these 
phases is further* broken down into several components, and flow- 
diagrams ar§ given which show, where the conponent's appear in the* " 
o\jjBrall process. The following, is a paraphrafsihg of Schrenk 's 
description of each of these components. 
-\ 

m Problem Recognition : - Determination th^t a problem 

requiring a decision ejtists. • • 

I " Acqu ire inf ormatio n: Receipt of information indicating 
that actual sltuatTon differs from the desired situation. 

Recognize obje^ctives ; The decision ^maker' s purpose or 
^ mission. 

• Perceive ^de cision need : Perception of difference between 

objectives and current situation; may result from change 
in situation or in objectives. 

- Assess problem urgenc y and impo r tance ; Establishrnerit of 
priority, of problem, relative to other ^problems demanding 
attention, and allocation of resources *for ^olving it. 

■ • ^ ■ - • ' 

# Problem Diagnosis : Determination of the situation that is 
causing problem. 

. Define p ossibl e situations : Generation of hypotheses 
regarding situation . 

* * 
Evaluate situation likelit ioods : Assignment of a priori 
, probabilities to alternative hypotheses.. 
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- Determine whether more information .is needed: Assessment 



of adequacy of information in hand; a continx^ng process. 

~ Identify possibl e datp sources ; If more information is 
desired."" ^ 

Jud^fe value versus cdst : To determine whether, or how, 
desired information should be acquired. 

~ Seek more i^nformation ; Assuming value judged to be greater 
than cdst. ' , , 

- Re-evaluate situation likelihoods t Iterate. 

' ' " ^ * 

- Determine whether alternative s u nder consideration account 
f6r all th e data ; Recognition oJ^' possible need to modify. 

• set of hypotheses^ being considered. 

-' Make diagnos tic decision: Selection of favored hypothesis^ 
or possibly of small se.t of weighted^ alternatives* 

Action Se lection: Choice of course of action. 
• *« 

Define action goals : Specification of explicit goals, 
including interim or sxlbl*rdinate objectives. . 

Specif y value and tim e cri .teria:> Identification of 
- relevant dimensions of mult i dimensional goals and 

specification of time constra^^rtts within which decision 
' mUst be made. \ , 

Weight decision cri teri a: Establishment o^ relative 
importance- of varibus decision criteria. 

~ Specify risk philoso phy ; Specification of strategy of 

action selection insofaaS as it is dictated by considerations 
of balancing risks against potential gains. 

Input operating d octrine; 'Considej^ation of any rules or 
doctrine by which the decision maker ^s behavior should be 
guided. 

GefieratgSyaction alternatives : Explicit listing of reasonable 
s^t oF~courses""oF action open to decision maker . 

- Pre dict possible outcomes: Specification of the possible 
outcome associated with each of the potential action 
alternatives. 

^ Estim ate outc o me ^ains and losses ; Determination of value 
of possTble decision outcomes. 

- Estima t e ou tcome like lihoo ds; Estimation of probabilities 
•of occurrence of possible outcomes for each action 

alternative. 
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- Evalua te expe cted -C^alu es df actions v ersus their costs : 
Derivation, from preceding two steps, ^ of expected value of 
each possible action, and estiiaatioii of associated *cost. 

A Evaluate actions >y ris]c philosophy; Assessment of each 

ajigtion alternative^ m terms of its implications for^-fche- ^ 

risk philosophy that the decision maker has adopted. / 

- Determine whether^ m ore info rmation j.s needed; As under 
Diagnpsis; a continuTng questioii;* new information might be 
useful either for identifying additional action possibili- 
ties, or to improve predictions concerning possible 
decision outcomes. 

- Seek informatio n; If desired, and worth cost of acquisition. 

" Re-evalu ate acti on alternatives; Iterate 

; ~ <f 

Determine whe t her best a ction is acceptable ; Review of most 

desirable actTon alternatTve to assure its acceptability, 

in terms of the decision goals and criteria, the expected 

gains from the choice and the cost of making 'it. 

- Choose course of action ; The -"deci^^ion. " 

" Iinplement a c tion ; Initiation 6f ^whatever steps are 

necessary Eo assure that the selected action is carried out. 

i 

^ The main fault that we have to ^ind with Schrenk's model is 
that it may be overly elaborate. It is doubtful that'many 
individuals go through anything approaching this multistep pro- 
cedure in the^ process of making a decision. This is perhaps an 
unjjastifie'd criticism, inasmuch as Schrenk intended the model to 
be wore prescriptive than descriptive. And whether such a model 
can serve as a prototype procedure for decision makers to follow , 
remains to be seenV /In any •case, the representation does serve the** 
useful function of making explicit many of the asppots df decision 
making and it stands reminder that decision making may be 

viewed, as a complex and multifaceted process indeed.N^ 

3 . 3 Decision Making as a Cpllect ion of Problem-Solvin g Tasks 

We take the position that decision making is best conceived" 
as a form of problem solving; or, more specifically, that it 
involves a variety of aspects each of which may be viewed as a 
problemy-solving task in its own right. In the most, general terras^ 
the decision maker's problem is to, behave in a rational, or at " ' 
least a reasonable, manner. Td be sure, the distinctive character- 
istic of the specific problems* with which tlje decrsion maker deals • 
is the element of choice; he must at some point decide upon one 
from among two or more alternative courses of action. While tl^e 
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act of choosing among alterna:tives' is centtal to decision making, 
-'It IS by no means the only problem—or even necessarp^ the most 
. difficult ,one--that the decision maker must solve<'''We wish to 
emphasize the .importance of. making explicit the bther things that 
must be done if one is.-motivated to make the best possible — or 
at least a satisfac1;ory-4decisioh^ given the resburces at one's 
disposal. In many real-life situations, the problem of choosing * 
among possible course^- of action is far simpler than that of 
discovering what one's options are inthe first plac^, or of 
assigning preferences to possi4)le decision outcomes in a con- 
sistent wa'Y.. ' Also, the decision maker may find it necessary to 
make many preliminary decision^ simply by way of setting the stage 
for making the decision' which is his primary doncern. Fpr example, 
he will want to reduce his uncertainty about, the^ decision situa- 
tion or al?out t^ie consequences of the various x:hoices . hat are 
■•open to him.?. However, the acqliisition of information takes time, 
* and may be costly in other v^ays, so he will continually be faced 
with the problem of deciding whether any additional information 
that he may wish to get is worth the cost of getting it. 

ft 

It is clear from the foregoing' that there are many ways to 
classify the various tasks that *the decision maker may be required 
to perform. The scheme that we fMid most satisfactory recognizes 
eight aspects o£' decision making: ^information gathering, data 
evaluation, hypothesis generation, problem ,s tincturing., hypothesis 
evaluation, preference specification, action selection, and ^decision 
evaluation. / 

' This cdnceptualization has an eaement of arbitrariness about 
— as does any other. There are four points that we would like 
to make in this regard. First, the decision to conceptualize the 
process in terms of eight type's of tasks,, as opposed to some other 
number, is itself somewhat arbi^trary, and reflects our own biases 
concerning what constitutes a useful level of organization. One 
might ^^conceptualize the decision process at a much coarser level 
and distinguish two major types of tasks — diagnosis and action 
selection — that would encompass all of those that we wish to 
distinguish. This approach has been taken by several investigators 
(Bowen, Nickerson, Spooner & Triggs, 1970), Kanarick, 1969; 
Williams & Hopkins, 1958).' Bowen et al. (1970) point out that in 
the milix^ary, diagnosis is, the proper function of intelligence, and 
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action selection that pf command, ^t the other extreme, one 
might attempt a much finer grained representation and ideritify a 
much larger number of activities that a decision maker may be 
called upon "to perform. In this case, each of the tasks we have 
identified might be replaced with several more detailed tasks. 
These are not mutually exclusive approaches, of course, and we will 
have occasion to consider how soyne of the tasks we ha\e identified 
may be further broken down". However, this level of analysis 
appears to^.us to be the most useful one for our present purpose, 
and possibiy for serving 'as a 'general framework in terms of which 
.to thinlc aKput. decision making as a whole. 

Second, *our taxonomy is not orthogonal, to other conceptuali- 
zations such as^ those discussed in the preceding section* It has 
elemeriUs in common, with most of them. Indeed, the intent is not 
to* take issue with other taxonomies, but to propose one that rep-- 
resents what, in our view, are the best aspects of all of them. 

\ ■ ■ ' ' • ■ ' 

Third, we ^o not mean to suggest that whenever aa^ individual 
finds himself performing . the role of a decision make? he explicitly 
rvms through this set of tasks in serial fashion, or feveH that he ^ i 
performs each of these tasks explicitly at a 1 1^- - Moreover, when 
he does perform these tasks it is not nejcessafily the case that he 
is fully aware of doing so. It is characteristic of human beings 
that they often can solve problems quite effecti\/ely* without having 
an'y bl^ear idea how they do it. This characteristic has b^een a 
frustration to researchers in artificial intellige.nce , w^ao have 
found it exceedingly difficult to program computers to perform some 
^tasks that human beings seem to be^ able to perform with ease. 
What we do mean to suggest by ^he proposed taxonomy is that all of 
these types of activities are implicated in decision making 'and 
that any attempt at a thorough dis-cussijbn of the decision-making ^ 
process must take account of them,. 

Finally, viewing decision making as a problem-solVing process 
that is composed of several phased or subprocesses^ emphasize^ the 
fact that in any given decision situation, different decision tasks 
could be performed by different individuals or groups (or Machines). 
An implication for training is that it ma^ be; less appropriate to, 
thirJc of training decision makers per. se than of training individuals 
to play specific roles in the decision-making pr6cess. On the other 
,hand, there will undoubtedly always^be some situations in which all 
the various aspects of a decision problem will be handled by the 
same individual. But whatever the case, there .is perhaps something 
to be gained by making decision makers — or specialist members of 
decision-making groups7-aware of the many facets pf the general task. 
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In the next few sections of this report, we consider each of 
the components of our task taxonomy in turn. The order in which 
the tasks are discussed tepresents a na'tural progression; however 
in real life decision situations, an individual, in a decision- 
making system, may perform several of these tasks more or less 
simultaneously. Or he may skip' from one to another in a variety 
of , orders, and may perform any given type of task many times in 
the course of attempting to solve a single decision problem. 
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SECTION IV 
INFORMATION GATHERING 

* 

From the point of view of the decision, maker, most decision 
situations are characterized by some degree of uncertainty. This 
uncertainty may involve the current "state of the world," the 
decision alternatives that are, availably, the possible consequences 
of selecting any given one of them, and even the decision maker *s 
preferences with respect to the possible decision outcomes. One 
of tbe majqr problems facing the decision maker , r therefore, is 
that of acquiring information in order to ifeduce his uncertainty 
Concerning such factors, thereby increasing his chances ^of making 
a decision that will have a desirable outcome. \ 

\ 

What makes the problem interestirtg and nontrivial, ,is the 
^f^ct that information acquisition can be costly, both in terms of 
time and money. Therefore, the decision maker must determine 
whether the value of the information that could be obtained through 
any given data-collection effort is likely to be greater than the 
cost of obtaining it. Aiid therein lies a decision problem in its 
own right. 

In theory, one can see an infinite regress here. In order to 
decide whether to initiate any information-collecting effort, one 
must determine the worth of the information to be collected and 
the cost of collecting it'. But in order to determine that, one 
may have to collect some information — at some. cost, and so on. In 
practice, of course, infinite regresses never occur; and in this 
case, one very quickly gets to a point at which the decision maimer 
rejies on information .in hand, or appeals ^to his own intuitions. 

4 . 1 Information Seeking versus Information Purchasing 

Information gathearing. may be thought of as involving two quite 
different activities: (1) information seeking (locating the infor- 
mation that ope needs ox wants) , and (2) information purchasing 
(deciding whether information, the location of which is known,, is 
worth v;hat it will cost to acquire it) . This distinction is some- 
thing of an oversimplification, inasmuch as the act of seeking 
itself typically involves some cost, and one often mustjflecide 
v/hether to incur that c^st without any assurance that the search 
will yield the information that is desireiS. .The aspect of "seek- 
ing" that we wish to emphasize, howevejr, is the need for identi- 
fying and actively searching out information sources, of finding 
opt where the desired information is and -going after i€. The tevm 
"purchasing" is used to connote a more passive role on the part of 
the decision maker, the opportunity to acquire information is pre- 
sented to him and he need only indicate whether or nqi. he wants 
to avail himself — at some cost — of the information that is* offered. 
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The distinction between information seeking and information 
purchasing is a useful one because it hrghlights the fact that 
experimental studies have focused almost exclusively on the latter 
process; although investigators o:^ten have not.made'the distinction 
and have frequently discussed their results as though they had to 
do with the foriper. Typicalljf, the decision maker is presented 
With all the information that he needs — although he may have to 
decide how much of it to purchase — and the process of seeking in- 
formation is not studied- The world outside the laboratory is, 
not nearly so accommodating, however, and one must either seek 
out the information one wants, or go without.it. Moreover, studies 
of information purchasing, while they tell us somethirtg -about how 
effectively pepple can judge the worth of information that is made 
available to them, shed little light, on information-seeking behavior. 

Perhaps the main reason why information-seeking behavior has 
not been widely studied is the difficulty of manufacturing situa- 
tions in the laboratory that are representative of those faced by 
decision makers in the real world. In any case, whatever the 
i-easons, information seeking per se has not received the attention 
from investigators of decision making. that it deserves. The ex- 
peiriments tha4 we have reviewed that purport to deal with this 
topic invariably have actually studied information purchasing as 
We have defined that term. 

4.2 O ptional-Stopping Experiments 

An experimental pa|adigm that has often been used, to study 
information-purchasing behavior is one in which the decision maker 
is provided with the opportunity on each trial either of purchasing 
more data that are relevant to the decision that he is required 
to make, or of making the decision. The terms "deferred decision" 
"optional stopping" and "optimal stopping*" have all been used to 
refer .to ^his paradigm. "Deferred decision" and "optional stop- 
ping" connote the fact that the subject in %uch an experiment has 
the option on each trial of making a terminal decision or deferring 
it invOrder to obtain more data. V'Optimal stopping" refers to 
the fact that when the situation is sufficiently well-stri:fctured 
so that the costs and payoffs associated with possible decision 
outcolnes, the cost and inf ormativeness of data, and the decision 
maker's objectives are all known, the point can be determined at 
which informai1:ion purchasing, should be stopped and the decision 
made. The "optional-stopping" paradigm' is to be contrasted both 
with the "more familiar paradigm in which the experimenter deter- 
mines how much informatipn' the decision maker will be given, and 
what is usually called the "fixed-stopping" paradigm in which the 
decision maker specifies how much information he wishes to purchase^! 
in advance of receiving any. ^ 
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Often, in optional-stopping experiments, the regnired de- 
cision concerns the paramete;t:S of the distribution Ltom which 
the observational data are being drawn. For example, one may 
have to decide whether a seqiience of red and black poker chips 
that one observes is drawn f:|:om a population in which the propor- 
tion of reds to blacks is, s^y, 60-40 or 30-70./ The question of 
interest in such experiments 1 is v/hether the subject's information- 
purchasing behavior deviates j from optima lity,. and if .so, in what 
way ? w j / ^ " ^ 

What constitutes optimal performance. has been worked out for 
a variety of specific situa^tions (Birdsall & Roberts, 1965; 
Blackwell & Girshick, 1954/ Raiffa & Schiaif er, 1961} • For our 
purposes it suffices to ]fecognize thaty in general, the amount of 
information (number of observations) that should be purchased , 
will vary directly with'^^the magnitude of the costs and values 
associated with the de(aision outcomes, and inversely, with the 
cost and "diagnosticity" of the data th^ are purohased. Diag- 
nosticity refers to the degree to^ which the data should reduce 
the decision maker *s uncertainty ^)out which of the terminal de- 
ci^on alternatives should be selected. The diagnostic value of 
a datum depends on' several factors (some of which are discussed 
in Section VIII), and typically decreases as the number of data 
that have already been collected increases, A factor that usually 
is not taken into consideration in optional-stopping experiments 
but' can be critical in real-life situations is the importance of 
time itself. In some situations the potential consequences of a 
decision are highly time-dependent. Thi^ fact can be incorporated 
in an optimal-stopping rule by making the cost of an observation, 
or the stopping criterion, a function of time. 

Typically, performance in optional-stopping experiments has 
been found not to be optimal. Moreover, as illustrated by a 
study by Green, Halbert, and Minas (1964), the deviation fi^om 
optimality may be in either direction. In one experiment. Green, 
et al. found that the number of observations purchased increased 
with the a priori uncertainty concerning the correct decision — 
as would be .expected of an efficient Bayesian processor — however:, 
subjects tended to purchase too many observations when the a priori 
uncertainty v;as maximized by providing no prior information con- 
cerning the likelihoods of the correctness of the possible 4e- 
'cisions. In combination, the results of these experiments suggest 
that decision makers may sometimes p*Urchase too much information, 
and sometimes too little. In particular, it would appear that # 
they may purchase too much information if the a priori uncertainty 
is small, and too little if the a priori uncertainty is large. 

Many investigators have used the optional-stopping paradigm 
(Becker,' 1958; Edwards, 1967; Ed\)ards Sr Slovic, 1965; Fried & 
Peterson, 1969; Howell, 1966; Irwin & Smith, 1957; Pitz, 1968; 1969; 
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Pruitt, 1961; Schrenk, 1964; Snapper & Peterson, 1971; Swets & 
Birdsall, 1967). The results of most of these studies suggest 
that althoug'h information seeking may approach optimal levels ^ 
(Becker, 1958; Howell, 1966; Pruitt, 1961), there are reasonably 
systematic departures from perfect performance. The general 
finding seems to be that too little .information is sought when 
(theoretically) much 'is required, and that too much is sought 
-when little is required. The la1;tei> finding fits well with the 
conservatism pr inertia effect often noted in studies of Bayesian 
inference, but the 'former clearly does not. 

A few descriptive models of optional-stopping behavior have 
been developed (see, for examples, Edward^, 1965a; pitz, 1968- 
Pitz, Reinhold, & Geller, 1969). These models have been developed 
in a Bayesxan conte^^t (Rapoport & Walls ten, 1972) and tend to be 
situation specific (se6, for example, the "World Series Model" 
of Pitz, Reinhold, & Geller, 1969). 

Noting that most optimal-stopping experiments had been con- 
cerned only with the question of when to stop acquiring information 
. from a single source, Kanarick, Huntington, and Petersen (1969) 
suggested that a more valid simulation of some decision-making 
situations f e.g., tactical situations, would recognize that the 
decision maker mu^st deal with information from more than one source. 
In keeping with this observation, Kanarick et ai. did an optional- 
stopping study in whicli the decision maker had the option on each 
trial of acquiring data from his choice of three sources,, or of 
making a terminal decision. The terminal decision that was re- 
quired involved the presence or absence of an enemy submarine in 
the vicinity. The information sources differed, both with respect 
to the cost of obtaining information from them and with respect to 
the reliability of the information obtained. (The topic of reli- 
abili^ of informatipn will be discussed more fully in Sections 
V and VIII.) .Costs associated with incorrect decisions were also 
manipulated. Although the behavior of the subjects was consistent 
with the rational model in many ways — they were willing to pay more 
for more reliable information; how much information they collected 
before making a particular decision depended on how bad the con- 
sequences would be if that decision proved to be incorrect — 
performance was less than optimal in several respects. The sub- 
jects tended, for example, to consult the most reliable (and most 
costly) sources less frequently and the less reliable (and less 
costly) sources more frequently than they should have. Kanarick. 
et al. characterized this behavior as a form of conservatism, "a 
reluctance to expend the resources necessary to obtain the best 
information In a choice situation" (p. 382). The subjects also 
tended to pifrchase less data in general than they should have, and, 
consequently, made more incorrect decisions and won fewer points 
than did a Bayegian model that was used to represent optimal be- 
havior . 
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Levine and Samet (1973) have also studied information gather- 
ing — information purchasing in our terms — in a simulated tactical 
situation. The scenario was a military action and the subjects' * 
4:ask was to decide which of. eight locations was the target of a 
hypothetical en^y advance. On each trial, a subject could either 
make a terminal decision or request additional information from 
each of three i'ntelligence sources concerning the present where-- 
aboutfe of the advancing force. A sequence of reports from a given 
source represented the path that the advancing force had taken 
over a period of time, According to that source. Among the vari- 
ables, that were manipaiated were the reliability of the inteUi- 
gence sources, the degree of conflict among reports from different 
sources, and the probability that a request for information would 
yield an updated report (as opposed to a repetition of the preced-' 
ing report). Performance was sensitive to each of the" variables. 
In particular, fewer reports were requested and 'decisions were 
more often correct when all thte sources were reliable, and the" 
quality of performance., tended to decline as the percentage of the 
sources that were unreliable was increased. Increasing the degree 
to which the sources were in conflict also had the effect of de- 
creasing the number of reports requested. (This counterintuitive/ 
result may be due in part to the fact that as conflict increased 
in this experiment, so did the probability that the correct target 
•v^as indicated by at least one of the sources on a given trial.) 
The number of requests for reports decreased as the probability that 
a given report would yield new information increased; the relation- 
ship was' such, however, that the amount of information (number of 
updates) received increased with this variable. ^ 

' In a subseauent e^xperiment, ,in which the same decision problem 
was used, Levine, Satnet, and Brahlek (1974) varied the rate at 
which reports were given to the subject, whether the reports 
were delivered automatically or in response to the subject's 
request, the possibility of revising an initial decision and the 
payoff scheme. In this case, performance was >etter for the faster 
rates of in:^ormation acquisition, but was not highly sensitive to 
whether the rate was self- or force-paced. Increasing the oppor- 
tunity for revising a decision had the effect of decreasing the 
accuracy of first decisions and ^he subjects' confidence in them. 

4.3 Decision^ Revision and Effect of Commitment on 
Information Gathering " 

The results of a few studies suggest that one's information- 
gathering behavior may be different after making a decision than 
before, particularly if the making of the decision involves some 
sort of public acknowledgment or commitment (Geller & Pitz, 1968; 
Gibson & Nichol, 1964; Pruitt, 1961; Soelberg, 1967). People may 
require more information, for example, to change a decision than 
v;as required to arrive at a decision in the first place (Gibson & 
Nichol, 1964; Pruiti:, 1961). This observation is in keeping with 
the results of several studies that suggest that evidence that tends 
to confirm a favored hypothesis is often given more credence than 
evidence that tends to disconfirm it (Brody, 1965; Geller-^ & Pitz, 
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1968; PitZ/ Downing, & Reinhold, 1967) • And sometimes disconfirm- 
ing evidence may even be misinterpreted as supportive of a decision 
that has already been made (Grabitz & Jochen, 1972)' • 
I ^ ' ' . . ^ . . 

The "motivation for acquiring infprmation may change, following 
a decision/ from that of trying to increase the probability of 
making a goocJ decision to "that of justifying or irationalizing' a 
decision that has 'already been made* Soelberg (1967) has concluded 
from a study of the job-seeking behavior of graduates of the Sloane 
School that people frequently make an implicit selection from among 
the existing opportunities, following which "a great deal of per^ 
ceptual and interpretational distortion takes place in favor of 
the choice candidate" (p. 29 ) • In, a somewhat similar vein, Morgan 
and Morton (i944) have a^sserted that people of ten ^^accept conclusions 
that are consistent with their convictions without regard for the 
validity of the inferences on which those conclusions are based, 
and that "the only circumstance under which we can be relatively 
sure that) the inferences of a person v/ill be logical is when they ; 
lead to a. conclusion which he h«is already accepted" (p. 39). We 
will return to the question of the logicality of thought in 
Section 8«3) . . . , 

One suspects that in real-v/orld situations the information- 
seeking behavior that follows the making of a decision may often 
differ considerably from that that precedes it* In particular, 
one would guess that to the degree that the motive of the informa- 
tion^ seeker is the rationalization of a decision already made, the 
process would become highly selective as to the sources consulted* 

4.4 Quantity of Information and _Q^uality of Decision 

% 

It is quite natural to assume that the more data one has that 
are relevant to a choice that he must make, . the better his choice 
will be. The assumption, without qualification, is not valid' 
(Ackoffy 1967; Fleming, 1970; Hayes, 1964/ Hoepfl & Huber^ 1970; 
Sidorsky & Housdman, 1966). It,, is possible, indeed easy, to provide 
an individual with more information than he can assimilate and use — 
especially if he is .operating under some time pressure. The point 
is illustrated nicely by an experiment by Hayes. ^ 

Hayes had naval enlisted men make decisions concerning which of 
several airplanes to disp.latch to investigate a reported submarine 
sighting in a simulated tactical situation. Tbe available airplane^ 
differed with respect to such characteristi-c-s as speedy distance of 
its base from the target, delay before it could take off, quality 
of its pilot, quality of its radar, arid so on. Each characteristic 
could take on any of eight (not necessarily numerical) "values," 
which could be ranked unequivocally from Ijest to worst. The number 
of available airplanes from which a subjeq^t had to choose was varied 
(4 or 8) as 'was the number of characteristics (2, 4, 6 or 8) on 
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which he was to base his choice. The effect of the latter variable ^ 
is of particular interest. Decision time increased markedly with 
this variable; however, the decision quality—which was defined ob- 
jectively in two ways— did not. Hayes hypothesized that, o^her 
things equal, one's sensitivity to the way two alternatives differ 
with respect to' individual characteristics decreases as the number 
of characteristics that must- be considered increases. Of particular 
relevance to this review is the fact that Hayes trained a second set 
of subjects for several days to see if they would learn to make 
better decisions with the larger amounts of information. Although 
the quality „of decisions was generally somewhat higher after training 
than before, the relationship between decision quality and number of 
characteristics on which a decisioa was based did not change. 

We should not conclude from this study that one should never, 
under any circumstances, be provided with more than a very few items 
of information that are relevant to. any choice that one may have to 
make. One might conclude > however : (1) that decision makers should 
be trained to recognize their limitations for assimilating informa- 
■tion, and to avoid attempting to operate beyond them, and (2) that 
to the extent that the functional relationship between the aesira- 
bilitv of the various choice alternatives that are open to the de- 
cision maker and the values of the factors that determine it known 
the implication of pa^rticular sets of factor values should probably 
be computed/ and not estimated by men. The problem of determining, 
or discovering, such functional relationships is a nontrivial one. 
(See Section IX.) 

• * /- ■ 
4-5 A Conceptuali2ation_ot_I^^ World 

What ^akes the real-world decision maker's task particularly 
difficult is the fact that the information that he would like to 
have typically is distributed among a variety of sources. One way 
of characterizing these sources is in terms of the two properties: 
- degree tOf passivity artd degree of cooperativeness. According to 
this conceptualization, a source is either active, or passive, land 
either cooperative or uncooperative. 

An actively coo£erative source— the-^pref erred 'type— volunteers 
informationr"and seeks"ways to get it to the decision maker. In 
the military context, an intelligence' offider would be an actively 
cooperative source for a coiiunahder. 

A passively cooperative source is one that would, provide in- 
formation if "solicltedr but does not volunteer it. A possible 
reason for not volunteering information in this case is .a failure 
-of the source to recognize itself as such. An example, again 'from 
a military context, wo.uld be friendly inhabitants of an area of 
operations who have information that would be valuable to a 
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military comm^nd^r, but are unaware of the fact. The problem that 
the decision maker has vis-a-vis passively cooperative sources is 
to identify and find them. 

An acti vely unco^qp>eratiY,e source has information that would , 
be of use to the decision maker ^ but being moMvated to thwart^ 
the decision maker's objectives if possible, volunteers information 
that is misleading. A propagandist is an example of such a source. 
The iSecision maker's problem witl^ respect to actively uncooperative 
sources i3 to recognize them as such and to assess the information 
obtained from them accoirdingly . * » 

^ pag sive ly uncooperative source is one that v/ithholds in- 
formation ' from the decision maker, and further will not pro\^ide 
it if .ask^d. Hostile noncombatants in an area of military operations 
might fit this de^scription, as might espionage agents. The decision 
maker's problem with respect to passively uncooperative sources i? 
to persuade them to change their status and become actively 
cooperative. History, both real and fictitious, is replete with 

V accounts of the unsavory methods^that have been employed to this 
end. ^ » 

• ■^ 

To the extent that . laboratory studies of decision making have 

^ been concerned with i/iformation gathering, they have involved 

actively cooperative sources almost exclusively. The problem of 

finding sources that are nonobvious and that of coping with those 

that are noncooperative have received very little attention from 

experimenters. In part this is undoubtedly due to the fact that 

capturing the essence of these aspects of information gathering in 

laboratory situations is a very difficult thing to do. And the 

alternative of s^tudying these processes in situ is hardly less 

difficult. Until such st:udies are performed, however, our under- s 

standing of how .decision makers go about gathering - especially 

seeking - information so as to increase their chances of making 

effective. decisions will remain very incomplete. . 

V* • . » 

4 • ^ i;^formation > Gathering and Training 

----- , J . 

We stress again that laboratory studies of information 
gathering have failed to capture the complexity of the problem 
that often faces the information seeker outside the laboratory. 
Consequently , very little, is known about information seeking 
behavior as it occurs in the real world. This is unfortunate 
% because information seeking constitutes a particularly critical 
aspect of many real-life decision problems and so long as this 
behavior- is not well understood, our understanding of decision 
making will be incomplete. 
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The implications for training ai;;e obvious: training procedures 
that are based on a solid foundation t^f f^actual knowledge about 
human capabilities and limitations cannot be developed if the foun- 
dation does not exist. The need is fo^ research that is designed 
to answer some of the questions that labt^ratory experiments here- 
tofore have, failed to address eff ectively\ Such questions include 
the following • How good are people at ideiitifying sources of 
information that is relevant to their decision problems? How do 
they gxj) about discovering ^uch sources? How ^capable are they of 
•assessing the cost of acquiring information that may be difficult 
to get and the worth of the information that m^ght be obtained? 
To what extent can useful principles and procedures for information 
seeking be made explicit and taught? It is probably fair to say 
that with respect to such questions there is insufficient basis for 
even an educated guess as to the answer. Clearly ^there is need for 
some imaginative research on this aspect of the decision-making 
process. 

Laboratory studies such as those reviewed above do shed some 
light on information purchasing behavior. In particular they tell 
us something ^out human capabilities and limitations in assessing * 
the worth of information ±n \Jell structured situations. Although 
it would be risky to generalize many of the conclusions uncritically 
to nonlaboratory situations, the conclusions nonetheless are sug- 
gestive of what should perhaps be done by way of training or train- 
ing research. 
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SECTION V 
DATA EVALUATION- , 

In the preceding section we used the words data and informa- 
tion more or less synonymously. It will be helpful at this point 
to make a distinc.tion. The term data is perhaps best used to 
refer to what one collects, afid the term information to connote 
whatever conclusions or inferences one draws from data. The data 
' and the information extracted therefrom can be identical, but .the^ 
need not be. For example, if a military commander ^receives data 
to the effect that the troop strength of an opposing tactdca-1 
force is 15,000 men, and he considers the source to be a reliable 
one, he will undoubtedly accept the data as accurate and conclude 
that the enemy troop strength is indeed 15,000 men* On the other 
hancJ, if he has less than full confidence ' in the sdurce of this 
report, he may tentatively conclude that the troop strt^agth is 
somewhere between 5,000 and 25,000 men, and attempt to get: more 
data'.from which he can derive a more precise estimate. 

•The point is that as part of the process of attempting to 
reduce- his uncertainty about his depisiOn situation, the decision 
maker must evaluate the (iata tl^at he receives as to their per- 
tinence and trustworthiness. In other words, the first decision 
^ that the decision 'maker must make^with respect to any* new datum 
is how seriously he should take it. He may not explicitly do 
this in all cases, but to fail to do so at least implicitly is 
tantamount to judging his sources as completely trustworthy and 
their inputs as equally important. 

« 

5.1 The Evaluation versus the Use of Data 



There are two questions relating to data quality that deserve 
attention: (1) how well can people judge and report the quality 
of the .data on which .decisions are to be based, and (2) how 
effectively can they- utilize information concerning quality of 
data when that information is provided for them? The first of 
these* questions concerns what we are referring to as the task of 
data evaluation, and is discussed in this section! The second 
has to do with data utilization and is more appropriately dis- , 
cussed in connection with hypothesis evaluation in Section VIII. 

In anticipation of the latter discussion, we note here simply 
that several experinlent? have bee^ addressed to the question of," 
how effectively decision makers use knowlejige of data quality. r 
In most sijeh studies the performance of subjects has^een compared 
with that of some ideal^ (usually Bayesian) model (see, for examples, 
Punaro, 1974: Johnson, 1974; Schum, DuCharme, & DePitts, 1973; 
Snapper & Fryback, .1971; Steiger & Gettys, 1972>. What is most' 
germane to the topic of this section* is the fact that the models 
that are used to represent optimal behavior typically 'distinguish 
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two separate steps. The first step entails an adjustment of the 
nominal diagnostic vaUue of a datum, the value *hat the datum would 
have if it"* were known to have been reliably observed or reported. 
The seapnd step involves the application of the modified datum to 
the hypotheses of interest. The first step is wHat^^we ar^ calling^ 
data evaluation, and it is important to note that the failure of 
subjects to perform this step properly appears to be one of the 
reasons why^they typically acquire less information from data that^ 
are not perfectly reliable than is there to be acquired. 

5.2 Studies of Data Evaluation 

Data evaluation has been recognized by the U.S. Army as. being 
of sufficient importance to warrant the development of a rating 
procedure for use by tactical intelligence personnel to evaluate 
all incoming "spot reports" (Combat Intelligence Field Manual, 
FM30-5) . . The procedure, which has been standardized for use by 
NATO army forces, re'quires that a sender of a report explicitly 
rate the report both with respect to the reliability of its source 
and the accuracy of its contents. The letters A through F are • - 
•used to designate estimates of reriability, and th4 numbers 1 , 
through 6 to represent judged accuracy^i The. first five ratings 
represent a scale going from "completely reliable" (A) to "un- 
reliable" (E) in one case, and from "confirmed by other sources" 
(1) to "improbable" (5) in the other. The lowest ratihg in each 
case is used to Indicate that^a judgment cannot be made^ "relia-^ 
bility'cannot be judged" (F) , "truth cannot be judged" (6). 

Obviously, the purpose of using such a rating procedure is 
to provide the receiver. of a report with some indication of how 
much confidence he should have in its contents. How effective the 
procedure has been, however^ is open to question. Data collected 
during field exercises have indicated that ratings often are 
omitted, from spot report^, and that the ratings that are use4 are 
too consistently high (Baker, McKendry, & Mace, 1968). The same 
study also revealed that the reliability' and the accuracy ratings 
tend to be highly correlated. One possible ^explanation of this 
correlation is that reliable sources tend to produce accurate 
'reports. This is an intuitively plausible explanation, and it 
raises the^questioTi of the need for two ratings. The other pos- 
sible explanation for the correlation is that the rater finds it 
difficu-lt to treat reliability and accuracy as independent dime^n- 
sions. The results of a subsequent laboratory study of rating 
behavior .wer$ interpreted as supporting the latter possibility 
(Samet, 1975b) • On the basis of his results, Samet proposed that 
an attempt be made to design and validate an improved procedure 
for evaluating intelligence data. Specifically^ he suggested! the 
possibility of assigning to a report a single number that would 
rjgpresent the evaluator's estimate of^the likelihood of the report 
being true, based on alL the. information available to him that 
was relevant to that judgment. * ' 
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5.3 Thfe Use of Nonquantitative Qualifiers ^ - ' 
* « • 

Probably most people who evaluate data or data sources do not 
do so according to a fonri^l procedure or irt quantitative terms • 
More typically, they use such qualifiers as "usually , reliable, " 
"not very dependable/' "prone to exaggerations^ "very precise," 
"a bit.careless,"-> "very likely r" "a roUgh estimate,-" and 'so forth. 
Swch ^phrases are certainly meaningful and und9ubtedly can convey 
important qualifying information.^ The problem is that not ^11 
people mean the same thing when they use one of ^hese phrases, and 
what complicates matters is the fact that even a given individual 
mgiy use the* same term to mean somewhat different • things at dif- 

f^ren^t times. . ^ ^ 

# * • 

*A number of effort^ have been made to measure the extent of 
agifeement between individuals in their use of such qualifying 
\ terms. A common experimenta^l paradigm is that of prQvi.'ing sub^ 

jects with lists of terms or phirases and requiring them, to trans-;- 
late the degree of certainty or uncertainty, denoted into ^ numeric 
(typically probaj^ili^stic) estimate. The variance observed among 
• and within subjects in the translatjLori then provides a measurement 
of agreement* Results o^ these studies (see, for example, Lich- 
tenstein & Newman, 1967; Johnson, 1973; Samet, "1975a, 1975b) 
typically show very low levels of agreeinent among subjects, and 
the potential fof considerable misunderstanding when large vo6abu- 
laries of qualifiers are, used. 

What factors influence the translation of a qualifier into 
a numeric estimate? TheVe seem to be no clear answers to this 
question. Cohen; Dearnley, and Hansel (JL958) suggested that con- 
text in which a word is used might play a ^ole, but a recent ^tudy 
by Johnson (1973) in which the encoding of 15 different probability 
words (or phrafses) contained in each of three different sentence 
contexts was explored failed to uncov^^ any significant context 
effect. On the other hand, a study ^ Rigby and Swain (1971) in 
which magnitude-denoting terms such as "couple," "lots,." and 
"bunch" were used did suggest such an effect. For example, a 
"bunch of missiles" had an average assignment ^of 7.73, while a 
"bunch of tents" had an average assignment of 12.32. It seems 
obvious on the face of it that nonquantitative terms denotihg 
physical magnitudes must be subject to enormous context effects. 
"Small" distances are measured in angstrom units by nuclear 
physicists and in light-years by astronomers. Indeed, it is 
difficult to see how, in the absence of context, such terms can 
be considered meaningful at all. Probability terms are difierent 
from magnitude terms in that prob.abilities are bounded whereas 
magnitudes are not. P^erhaps this helps to account for the former's 
greater independence of context. * It shtDuld be noted that neither 
Johnson nor Rigby and Swain found significant dir f erence? in the 
use of these terms due to group membership (arr/y enlisted men and 
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graduate students, in the former study; army helicopter, Air Forc^ 
prop, Air Force jet, and Navy attack bomber pilots in the latter) . 

5^4 Data Evaluation and Training , 

It seems clear that full exploitation of computer-based tac- 
tical data-analysis systems will ultimately require the use of 
numeric values in place of qualitative estimates, if the relia- 
bility of data is to be taken into account when they are used. 
How best to ^^^rive at these values is, at this point, a matter of 
conjecture. One could attempt to establish a formal vocabulary, 
of qualitative terms and phrases, associate with each term or 
phrase a specific numerical v^lue (or range of values) , and train 
personnel to use the resultant ispmorphisms in encoding and de- 
coding communications • This isythe esse.nce of a proposal made 
some years ago^by. Kent (see Pl^ftt, 1957) • Considering, however, 
that formal training would be^a requirement in any ca;.v^, a pre- 
ferred alternative to this approach is to instruct dec i^ion^ makers 
in the use of probability (and magnitude) scales and require 
estimates to be pommun-icated in explicitly quantitative terms 
(Johnson, 1973; Samet, 197^).* Th^ obvious problem for training 
research is that of developing effective procedures for trailing 
people to evaluate data quantitatively and for increasing the 
intra-r and inter-person consistency with which quantitative 
assessmeiits are made. 
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. • SECTION VI * • 

PROBLEM STRUCTURING * 

An exceedingly important step, in solving anv problem is to 
be-guii:e expl-i-c±t about what t^Tie problem is that" one is to solve. 
And one way to be explicit is to attempt to represent the problem 
in terms of a formal structure. While the need to be explicit 
may appear to be too obvious to deserve comment, it is also ap- ' 
parent that satisfying that need is, not always ^n easy thing to do 
Attempts to apply computers to problem-solving tasks have high- 
lighted both the need for explicitness and the difficulty of ob- 
taining It. Armer (1964) has commented on the frustration that 
IS sometimes entailed when one tries to formulate a problem in 
such a way that a comput,er can help solve it. He illustrates his 
point With re,ference to a bank official who stated, after having 
his banking procedures mechanized "that 65 percent of the data- 
processing -group' s, -effort went to deciding in detail wh t problem 
they were solving" (p. 250). Presumably, the investment -as worth 
it; without It, they could not have recognized a solution had thev 
found on^. • 

" • - . 
The. act of trying to make the structure of a problem explicit 
can be an instructive experience for a problem solver, inasmuch ~ 
as it forces on him the realization of what he does and does not 
know about the problem on which he is working— or thinks he is. 
Essentially, this observation is made by .Cloot (1968) vis-a-vis 
the application of computers to the decision problems of manage- 
ment. He takes the position that one of the tt'^jqx benefits that 
is to be derived from an attempt to implement a computer-based 
management information system is not the help that one would get 
from a functioning system," but what one can learn about the prac- 
tice of management from the implementation effort.. "it can even 
be- argued' that the successful use of a conjputer-based fti's should 
be measu-red by the ext'ent to which managers learn to improve 
their performa^ice so that they can discard it again... There is 
no doubt that the changes that do come about will be due more to 
managers having a better understanding of thei. decision processes, 
than to the technical facilities of the computer" (p. 280) . 

■ A major contribution of theoretical treatments of decision 
making is the provision of formal models in terms of which a de- 
cision maker can atteijipt to structure his own decision problems 
Invariably, such mc)d els are simplified abstractions, and conse- 
quently may not do justice to' the full details of any given situa- 
tion.. Nevertheless, they do provide one with structured ways of 
viewing things, . which may make the problems easier to think about, 
and as a consequence— hopefully— easier to solve. It has been 
suggested that this is the way in which quan;:itative models will 
have their primary effect r "I believe that the greatest impact of 
the quantitative approach will not be in the aria of proSem Jo?- 
vina, although it will have growl^ usefulness there. ??s grel?ist. 
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impact will be on problem formulation ; the way managers think ' 
about tl:^eir problems —how they size them up, bring new in^sights 
to bear im them, and gather information for analyzing theih. In 
this sens^, the results that 'quantitative people'* have produced 
are beginning to contribute in a really significant way to the 
art of management" (Hayes, 1969, p. 108). 



6.1 State-Action Matrices 



Probably the most well-known way of representing decision situ 
ations is in terms of state-action matrices, ^ Such matrices make - 
three aspects of decision situations explicit': the hypothesized 
possible "states of the world," the action alternatives that are 
open to the decision maker, and the decision maker's preferences 
with respect to the various possible state-actidn combinations.' 
Sometimes such mattices are referred to as payoff matrices inasmuch 
as each cell of the matrix represents the cost or val'^e — or utility 
— decision maker of the outcome of a Rarticulai action se- 
. lection, given that the associated state hypothesis is true. A 
decision problem may be represented in this wa^ as follows: 

Action Alternatives 

■^1 ^0 • * • A . ' . . . A 

_± £ 1 n 



Hypothesized H^ U^^^ U^2 

States 

of the "2 "21 "22 

World 



H. U. . 

^ ID 



H U ' 

in ' mn 



Much of the theoretical-analytical work on decision making 
has been concerned with optimal strategies for selecting action 
alternatives once the situation has been formally structured. 
Given an explicit decision gpal (e.g., minimization of risk, 
maximization of expected gain ^ and a formal representation * 
of the situation, prescriptive models can provide useful guidance \ 
for action selection. The process of representing real-life 
decision situations formally, however, is at the present time more 
of an art than a science. Examples of decision situations that 
are easily structured can always be found; however, not all de^ 
cision problems can. readily be forced to fit the same mold. 
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^ Ev^n given that the structure shown above is an appropriate 

one for a particular problem, it is clear that -in order teJ use it 
one must be able to specify, as a minimum, what the hypolhesized 
states of the world are, what one's action options are, and how 
-the various possible decision outcomes (state-action pairs) relate 
to tl\e value system that will determine the desirability of the 
actual outcome. One may find it necessary to engage in a consider- 
able amount of information seeking in ordfer to fill out" such a 
structure. Moreover, how one does fill out the structure is deter-- 
mined in part by exogenous variables over which one has no control, 
and in part by self --imposed constraints. The state of the world 
tends to be beyond one's control; all one can do is atteiript to 
determine what it is likely to be. One's action alternatives/ 
however, may be constrained in part by limits that are self-imposed. 
What are viewed as viable strategic military options, for example,, 
may depend^^n the particular military doctrine in vogue at the time. 
Benington (1964) points out that the ba^ic concept behuid the 
development of such automated, or selniautomated, systems -s the 
SAGE system in the 1950 's was the concept of "s^t-piece warfare." 
"Set-piece warfare is characterized 'by warning of threat, total * 
and preplanned goals, speed of. response, and detailed and precise 
management of the campaign" (p. 9). Emphasis is on massive re- 
taliation totally preplanned, or "spasm" response. During the 
early 19550' s, the set-piece warfare idea lost favor. President 
' Kennedy and Secretary of Defense McNanjara began to emphasize the 

-importance of flexibility and adaptability, the' ability to make 
selected and controlled responses, directed toward military (non- 
civilian) targets and appropriate to the (not always foreseeable) 
contingencies that elicit them. Clearly, the set of agtion alter- 
natives that the strategist will consider .under one of' these re- 
taliation doctrines is quite different from that that he will 
consider under the other. ) 

^•2 Alternative Structurings of a Given Situation ^ 

^ It is apparent that to think in terms of the structure of a 
decision space is to oversimplify matters greatly. Usually any 
given situation can be structured in a variety of ways. Moreover, • 
how one chooses to represent ^a particular situation* may not be ' ' 
incidental. It seems to Be true of problem solving in general 
that how one represents a problem can be an important factor in 
determining how easily one can then solve it. This point has often 
been made by individuals engaged in efforts to program computers 
to perform intellectually demanding tasks (see, for example, 
Nilsson, 1971). The same probl^ may yield to -"attempts to solve 
it when- represented in one way while resisting such attempts when 
represented in anbther. s t 

An important aspect of developing a useful structure is that 
of conceptualizing a situation at an appropriate level of detail. 
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Too simple a structure may violate the comple:jcity of an actual 
situation. 6n the o€her hand^iCharles Pierce's maxim that "a few 
clear ideas are worth more than many confused ones" seems particu- 
larly apropos .J[iere, We . state as a conjecture that a necessary 
requisite for effective 4®<^ision making is the ability to get 
quickly to tljjxe heart of a problem, to concentrate on. essentials, 
and to ignore irrelevancies. What this often* means ip practice is 
being able to see. through superficialities that frequently obsqure' 
underlying issues. Moreover, even when the' situation, stripped of 
incidentals, is ii>Jierently complex, there may be some meri^t in a 
simplified conceptualization of it, provided that the fact that the 
Qonceptualization is a simplification is not then promptly forgotten. 
There is little to be gained by representing a situation i^ such 
a cofUplex way that the decision maker cannot grasp the representation 
intellectually. What constitutes an optimal level of detail may 
vary from situation to situation and from individual to individual, 
but variability in this regard may not bp very great. We suspect 
that for the vast majority of situations and decision me^'-^rs a 
representation that involves more than eight or ten hypothesized 
states of the world and as many action alternatives, at any given 
level of 'description, will prove to be an unwieldy^ one. 

6.3 ' Structuring as an. Iterative Process 

dn the basis of an anlaysis of protocols obtained in h$s 
classical study of problem solving, Duncker (1945) reached a con- 
clusion that is gerr/ane to the issue of problem structuring. The ^ 
problem that he used most frequently in his studies was the now 
well-known radiation problem: "given a human being with an inoperable 
stomach tumor, and rays which destroy organic tissue at sufficient 
intensity, by what procedure can one free him of the tumor hy^^ these 
rays and at the same time ^void destroying the healthy tissue which 
surrounds it?" (p. 20 ) .. The conclusion that Duncker came to after 
observing the efforts of many people to solve such problems was 
that the development of a solution typically proceeds frbm the more 
general to the more specific. (On this point, see also Hogarth, 
1974/ and Kleinmufez, 1^6jb.) Th^ principle by which the problem 
is, hopefully, to be solved etnerges first, and the deb^ils of the 
solution come later. Lt o*ften happens that a principle may be 
valid, but there turns out to be no feasible way tc/ implement it. 
A jirinciple that was frequently identified in the case of the radi- 
ation problem, for example, was "avoid contact between rays '^nd 
healthy tissue." When the problem solver could think of no way to 
do this and still get the rays to the tumor, he had to abandon the 
principle itself— even though it was a sound one — and search for 
another that was not only sound but practicable. 

The finding of a new principle, or a general property of a 
solution, always involves, Duncker suggests, a reform\>lation of 
the original problem. In the case of the example just given, having 
accepted "^voiding contact" as a valid principle, one has in effect 
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defined his problem as that of finding a way to do just this. When 
forced to reject ^ given principle as impractical, the substitution 
of another (e.g., "lower the intensity of the rays on their v/ay 
through healthy tissue") in effect defines another liow-to-do-it 
problem to be solved. "We can accordingly describe a process of 
solution either as development of- the solution or as development 
of the problem. Every solution-principle found in the process, 
which is itself not yet ripe for concrete realization. • . functions 
from then on as reformulation, as sharpening of the original setting 
of the problem. It^is therefore meaningful to say that ^what is 
really done in any solution of problems consists in formulating the 
problem more productively . To sum up: The final* form of a Solution 
is typically attained by way of mediat^ing phases of the process^, 
of^ which each one^ in retrospect j possesses^the character of a 
solution:, and J , in prospect^ that of a problem" (p. 34, italics his). 

It is> probably th^ case that complex decision problems, like 
other types of complex, problems, yield grudgingly to atteiapts to 
structi^re them. Moreover, a decision maker may find it necessary 
to^ formulate and reformulate a decision space several times before 
arriving at a structure that^ he feels adequately represents the 
decision problem that he must solve and dlipes so in a way that 
^facilitates arriving at ^ solution. The willingness to discard 
a favored conceptual framework when it is seen no longer to fit 
the facts in hand has been considered by some to be one of the 
defining characteristics of original thinking (Mackworth, 1965; 
Polyani, 1963)-. - ' ' ' 

6 . 4 Problem Structuring and Training 

The question of how to train decision makers to structure 
decision problems effectively has received very little attention. 
Moreover, if it is true, as Edwards (1973) has suggested, that of 
the several aspects of decision analysis the process of problem 
structuring is least amenable to formal prescription, exactly what 
should be taught is not clear. 

It seems likely, however, that something is to be gained by 
familiarizing decision makers with such formal representations-- 
models — of decision situations, as are provided by decision theory 
and gamp theory. Such training should be conducted in such 'a way 
as not to leave the student with the unrealistic idea that all 
decision situations are readily, represented — without distortion — 
by the same model. 

, Practice in representing specific situations in terms of such 
models, and dfiteria for judging the relative merits of different 
models for different problems should probably be part of any 
training program in decision making. Practice in representing a 
given decision problem at different levels of detail also would 
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probably be beneficial. Duncker's work suggests that one approach 
to problem structuring that might usefully be taught is that of 
zeroing in on /an appropriate formulation by a 'series of approxima- 
tions , proceeding from the more general to tlie iQore detailed. 

But thqse are only conjectures. The fact i/S that little is 
known about how to train a person to be good at imposing structure 
on a problem — whether it be a decision problem or a problem of 
any other kind. Mackw6rth (1965) has noted that one of the char- . 
acteristics of creative individuals is an exceptionally strong 
need to find order where none appears on ^the surface. If this 
is so, then one way to train people to be fetter problem structurers 
is to train them to be more creativev If only we knew how to do 
that! . ' X * ' 

An alternative to training decision makers to formalize their 
decision problems is to. provide them with models that cire appro- 
priate to their particular, situations, and that can then be used 
as decision aids. ^tns?:y (1970) hafe suggested this possibility. 
A model that is to be used by a decision maker need not be genera- 
ted by him,* but, Gorry points put, ^ it may be derived from his 
description of the situation, and it must be thoroughly under- 
standable by him. In this case the training task becomes that of 
teaching an individual to make effective use of the structure that 
someone, else ^has imposed upon his problem. ^ 

At least one study has been addressed to the question of the 
subtasks in terms of which one class of decision makers sees deci- 
sion-making and how this view would change a result of training. 
Hill and Martin (1,971) gave secondary-school teachers problem- 
golving exercises designed to- train them with respect to some of 
nineteen specific skills that they associated with decision making 
and to acquaint them with a particular model of the decision-making 
process (see Section III). Both before and after training, 'the * 
subjects were asked to list the specif ic> steps that they would 
take in an effort to solve a hypothetical prcble^n involving an inter 
person conflict. t>erhaps the most striking aspect of the results 
was how large a proportion of the steps that subjects^ listed fell 
in the "formulating-action-alternatives" category. Before* training / 
more of the listed steps fell in this category than in^the other . 
five combined. The main effect of training was to reduce^ the num- 
ber of steps in this category by about two-thirds and to increase 
the usage of some of the other categories slightly; but formulating 
alternatives still remained the largest category. The investigators 
concluded that training had made the participants more avare of the 
several activities involved in decision making, but pointed out 
that their results shed no light on the question of whether much as 
increased awareness . v/ould produce better decision making. 
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SECTION VII 

HYPOTHESIS GENERATION 

"Hypothesis ge^^^^ doseiy associated with problem 

structuring. We find it convenient to consider it separately, 
however, because it is a more narrowly focused type of activity.. 
Problem structuring is always important. Even when complete in- 
formation is available concerning the state of the world, the 
action alternatives and all the possible decision outcomes, it is 
still necessary to cast the problem into some mold, and the mold 
that is chosen may have much to do with the decision that is made. 
I^ypothesis generation, on the other hand, is a necess.ary activity 
in those decisiqn situations characterized by uncertainty about 
such things as the state of the world and the implications of 
selecting specific' decision alternatives. Often, in spite 
of one*s best efforts to gather information, it is not possible 
to eliminate uncertainty about these things completely: In such 
cases, it is convenient to conceptualize the decision 'ma];or' s view 
of the situation as a set of conjectures, or hypotheses, 

7.1 Hypothesis Generation versus Hypothesis Testing 

Investigators of cognitive processes have long recognized 
two rather different types of thinking, Bartlett (1958) speaks 
of closed versus adventurous thinking, Guilford (1963) of -conver- 
gent versus divergent thought, Mackworth (1965) distinguishes 
problem solvers and -problem finders, ^he one kind of thinking 
tends to be deductive and analytical; the other inductive and 
analogical. The first has to do with evaluating hypotheses, the 
second' with generating them. The history ^of science attests to 
the fact that the ability to evaluate hypotheses, to deduce the 
implications of theories and put them to empirical test, is a far 
more common quality among men than is the ability to generate 
^ hypotheses, to construct theories that organize and structure 
facts that were not perceived as related before. 

Some forma'l treatments of decision making require that the 
situation, as viewed by the decision maker, be conceptualized as 
a ^et of mujtually exclusive and exhaustive hypotheses, each of 
which represents one of the possible states of the world. As da^a 
are gathered, they are used to modify a set of probabilities, each 
of which represents the decision maker's estimate of the likelihood 
that a given hypothesis is true. Much laboratory experimentation 
has been -devoted to the question of how effectively man can assi- 
milate 'Sata and use it to modify his view of the world as implied 
by the probabilities that he associates with the hypotheses that 
he is entertaining. (We will consider that problem in the fol- 
lowing section.) However, very little attention has been given 
to the question of how* capable people are of generating a reason- 
able set of hypotheses to begin with/ or of modifying the set 
when the need to do so arises. 
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Typically, all of the hypotheses that are to be considered 
are provided for the decision maker in advance, so the process of 
i^ypothesis 5^^^ idoreover, formal decision 

procedures usually permit the decision maker onl y to update the 
probabilities that have been assigned to the previously established 
set of hypotheses/ They fail to recognize the fact that it may 
be the case in real-life situations that a set of hypotheses that 
is originally developed may not contain the hypothesis that will 
eventually prove to be the true t)ne. It ofteh occuts in real-life 
situations that incofaing data suggest to the decision maker new i 
hypotheses that have not yet been considered. Any decigionrmaking 
procedure that purports to be generally valid must provide for 
establishment of new hypotheses .whenever the information in hand 
indicates the need for them. 

7.2 Importance of Hypothesis Generation 

The importance of the function of hypothesis genera cion can 
hardly be overemphasized. To be sure, one may think of some de- 
cision contexts for which all the potentially interesting hypotheses 
can be specified in advance. For example, it may be the case for 
some straightforward troubleshooting situations that an exhaustive 
set of the hypotheses of interest can be* listed prior to the per- 
formance of any tests. More typical of complex decision prob.lems, 
however, is the case in which the set of possibilities is either 
not fully known, or too large to b6 listed exhaustively. The 
problem of the physician who is attempting to diagnose an illness 
with a set of . symptoms that does not fit a common pattern,, or the 
investor who is trying to gauge the risks and potential gains in 
a speculative financial venture, or the computer programmer who is 
tracking down an elusive bug^ or the tactician who is trying to 
assess the significance of some unorthodox behavidr on the part 
6f a wily opponent is less that of testing prespecified hypotheses 
than that of defining hypotheses that it would make sense to con- 
sider. / 

^ The difficulty is not so much that of representing a* decision 
situation in terms of a set of possible states of the world that 
is exhaustive and mutually exclusive. The problem is that of 
coming up with a set of possibilities that is useful from the 
decision maker's point of view, A military commander can always 
represent the alternatives that are open to an adversary in terms 
of such gross action categories as attack, defend, and withdraw, 
and the ability to distinguish among these' possibilities would 
undoubtedly be of interest. However, a commander's decision-making 
responsibilities typically require much more precise information 
than would be provided by the resolution of the uncertainty implicit 
in these three possibilities. That is to say, he wants to know not 
only whether^ enemy forces plan to attack, but at what time, in what 
strength, at what locations, and so forth. It is at this level of 
representation that the command:er's (or perhaps his intelligence 
officer *s) hypothesis-generation capabilitj.es 'are put to the test. 

49 63 
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7.3 Experiments on Hypothesis Generation ' 

The study of hypothesis generation in the laboratory has 
, often involved "concept attainment" or "discover the rule" type 
tasks. The work of Bruner, Goodnow, and Austin (1956) illustrates 
the use ''of concept attainment tasks to siudy this aspect of think- 
ing. In a typical experiment, a subject attempts to identify a 
concept that an experimenter has in mind. Tl^e concept usually is 
defined in terms of conjunctions or disjunctions oZ specific stim- 
ulus attributes (e.g./ "red and square"; "blue or yellow, and not 
circular"). In some situations the subject is shown stimuli, some * 
of which belong to the conceptual category that he is attempting 
to identify and some of which do nbt. He is told which stimuli 
are which and from this "exemplar" information he is to attempt to 
identify the concept. Sometimes the subject chooses the stimufi 
that he sees, in which case the task can also be used to study a 
form of information-gathering behaviqr. 

Obviously, the performance of this task involves hypothesis 
testing (a topic to which we will turn in the following section) , 
but the key problem it that of hypothesis generation. Unless one 
comes up with the right hypothesis to test, ;fche testing that he 
does will only eliminate some of the untenable possibilities, of . 
which there may be many. 

A basic conclusion that Bruner et al. draw from their experi- 
mental results is that the strategies that subjects employ in these 
, sorts of tasks can be isolated and described. They identify four 
such strategies, for example, that subjects^ 'u3e when they have the 
job of discovering a conjunctive concept by selecting stimuli and 
being told, concerning each stimulus selected, whether or not it 
is an exemplar of the concept that they are attempting to identify. 
These strategies differ in terms of the balance they strike ambng 
three parameters: the amount of information obtained from an ob- 
servation, the cognitive strain imposed on the subject (amount of 
information that must be carried in memory, extent to which involved ' 
inferences must be made), and the risk that the strategy will fail. 
'The strategies are defined in terms of the nature of the hypotheses 
that are generated and put to the test. In one case, for example 
("successive scanning"), one specif ic' concept is hypothesized at 
a time, and stimuli are chosen in such a way as to test that hy- ' 
pothesis directly. In another case ("conservative focusing"), 
the initial hypothesis, in effect, includes several possible con- 
cepts and an attempt is made to discover the defining attributes 
systematically one at a time. Which of the several strategies is 
most appropriate depends on the details of the experimental situa- 
tion. 

Bruner et al. found that the strategies that subjects use 
tend tcvchange appropriately in response to changes in the 
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experimental situation; and, on balance, these investigators con- 
sidered the performance of their subjects to be quite good. In 
their words: "in general, we are^-struck by the notable -f-lexibiliirF 
and -intelligence of our subjects in adapting their strategies to 
the information, capacity, and risk requirements we have imposed 
on them. They have altered their strategies to take into account . 
the increased difficulty of the problems being tackled, choosing 
methods of information gathering that were abstractly less than 
ideal but that lightened pressures imposed on them by the tasks 
set them. They have changed from safe-but-slow to risky-but-fast 
strategies in the light of the number of moves allowed them. They 
• have shown themselves able to adapt to cues that wfere less than' 
perfect m validity and have shown good judgment in dealing with - 
various kinds of payoff matrices. They have shown an ability to 
coiT^bine partially valid cues and to resolve conflicting cues" 
(p • 23 8 ) • 

* 

. Performance was not ideal, however.' Among the limitations ' 
that were .noted were a tendency to persist in focusing on cues 
that had proved to be useful in the past even if they were not 
useful in the present, and an inability to make as effective use 
of information gained from noninstances of a category as of that 
gained from category exemplars. 

-Bruner et al. also found that concepts defined in terms of ^ 
disjunctions of stimulus attributes were more difficult to discover 
than those that were conjunctively defined. This finding has been 
corroborated by Neisser and Weene (1962) who used a large variety of 
attribute-combination rules. Not surprisingly, concepts defined 
m terms of the presence or absence of a single attribute are 
easier to attain than are those defined in terms of conjunctions 
or disjunctions of two or more attributes, which in turn are 
easier than those defined in terms of more comglex rules involv- 
ing combinations of conjunctions and/or disjunctions (Havqood 
& Bourne, 1965; Neisser & Weene, 1962). 

Another experimental' task that has been used to study hypo- 
thesis generation is that of discovering the rule by which a 
specific sequende of numbers or letters was generated. Typically, 
the subject is shown one or more sequences (or segments of se- 
quences) that satisfy the rule. He then can propose other se- 
quences, or continuations of the segment, in order to test the 
validity of tentative hypotheses that he may wish to consider. 
Each time he proposes a possibility he is told whether it satis- 
fies the rule; and when he feels he has obtained enough information 
to justify doing so, he is to state the rule. 

Again, performance of this task obviously involves information 
gathering ahd hypothesis testing as well as hypothesis generation, 
but hypothesis generation is in some sense central, what information 
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is sought is likely to depend strongly on what rule is being con- 
sidered.. Mof^sover, unless the correct rule is hypothes^ized at 

--.some -point-r"it-oannot--be^-test:ed" arr^ - - - 

The results of experiments along these lines have revealed 
some^ interesting deficiencies in hypothesis-generation behavior 
which appear to stem fxom a lack of understanding of some basic ^ 
rules of logic. Wason (1974) has*^ described some results that 
suggest that people may have particular difficulty in discovering 
rules that are sufficiently ^general that they subsume many rules 
that are more specific. For example, the rule "any three numbers 
in increasing order of magnitude" proved to be particularly dif- 
ficult for his subjects to discover. If, as examples of triads 
that conform to this rule, a subject were given (8 10 12,), (14 
16 18) and (20 22 24), he might quickly generate the hypothesis 
"successive even numbers," test it with other sequences that 
satisfy it, and then announce this rule with confidence. What is 
disappointing about this behavior is the failure to Tiypothesize 
alternative rules to which the given sequences also conform, and 
then to consider sequences that would discriminate between the 
alternatives hypothesized • More disturbing, however, is the 
finding that even when told. of the incorrectness of a hypothesis, 
and presented with conclusive infirm evidence, subjects some- 
* times insisted that their hypothesized rule was validated by the 
fact that all the test sequences that they generated conformed 
to it. 

• 0 

Two other results noted by Wason are relevant to the problem 
of hypothesis generation r because they also demonstrate hQw the 
, process can g^t bogged down. First is the possibility of perse- 
veration with an invalidated hypothesis without recognizing that 
one is per sever a):ing. He notes, in this regard, that what subjects 
often do when informed that a hypot^jesized rule is not the correct 
/ one is to generate additional triads' that are consistent with that 
rule and then announce* the same rule expressed in different terms. 
Second is, a tendency, when hypothesized rules are invalidated, to 
generate more and more complex rules rather than simpler , ones. 
The following example is' given of a third generation rule produced 
by one subject: "The rule is that the second number is random, 
and either the first number equals the second minus two, and the 
third is random but greater than the second; or the thitd number 
equals the second plus two*,, and bhe first is random but less than 
the second" (p. 382) • Recai:j. that the correct rule was "any, three 
numbers in increasing order of magnitude." One Jorlclusion that 
may be drawn from this type of experimental finding is that the 
discovery of a general rule, even though conceptually simple, may 
be imt)eded by the discovery of more specific rules whos^ exemplars 
are also exemplars of the more general rule. 

* 
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7,4 Hypothesis Generation and Training 

Hypbthesis generation represents the same sort of challenge 
to training and training research as does problem structuring. 
•The basic n4ed in both cases is for a greii;er understanding of 
how to promote creative thinking. * . ^ • 

> / 

A specific problem that deserve^ attention from training 
specialists is that of perseveration.. Results such as those ob- 
tained by Bruner, Goodnow, and Austin (1955) and by Wason (19 74) 
indicate the need for training pjjocedures designed tp improve 
the ability, or increase the willingness, of decision 4nakers to 
generate alternatives to the hypothesis, or hypotheses, under 
consideration. They demonstrate the importance of sensitizing 
decision makers to the danger of accepting a hypothesis on the 
basis of insufficient evidence r and to the fact that the best way 
to avoid this mistake is to attempt to generate plausible alter- 
natives and to seek the kind of data that will be most likely to 
discriminate among them. , , 
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SECTION VIII 

^H-y-pe'XH&srs -evaluatxon - 

4 , ' * 

Narrowly defined, hypothesis evaluation refers to the process 
cf applying data to the assessment /of the likelihoods of one's . 
hypotheses concerning the unknowns^ of the situation. More generaly, 
the term might be used to connote the process of extracting^informar 
tion from data, of attempting tt) reduce one's degree of uncertainty 
, about the parameters '^of the decision space. ^ In some formally struc- 
tured approaches to decision making, hypothesis evaluation may involve 
the revisions #of numerical probability estimates qr other/ quantita- 
tive indicants of relative liJj&elihoods. In other cases the process 
may be less explicit, but it is not for that reason less important. ' 
We kssume that even in situations that hav^e been given little formal 
structure, the decision maker attempts to make use of au least some 
of the data that are available to him, in order to clarify his vi^ew, 
or perhaps to confirm his assessment, of the situation. - 

The following discussion takes a rather broad vipw of % 
hypothesis evaluation. It touches on a number of topics that 
relate to man's abilities, limitations, biases and predllectipn^ 
as a processor of information or a user of evidence. In some cases, 
it may appear to range beyond the specific subject of hypothesis, 
evaluation, and^deal with "thinking" more generally. Our reason 
for including this material is that it seems to us tifelevant to .ithe 
problem of decision maJsJing, and it appears to fit ijio're readily hdre 
than elsewhere within our conceptual framework. In Section 8.6, 
the discussion b&comes ^^^^rrowly focused on the problem of revising 
probabilities in situations that have been formalized to the extent 
that a Bayesian dSta-aggregation algorithm might be applied. 

8.1 Serial versus Parallel* Pro^Qssing^ ^ 

One qu'festion of interest concerning the vay people evaluate^ 
hypotheses is whether they consider th,em one, or several, at a 
time. Empirical data are lacking on the question of which of 
these alternatives best characterizes man's approach to hypothesis 
evaluation. It is our impression t^a,t the prevailing consensus is 
that the assumption of seriality is the more plausible of the two, 
insofar as the conscious consideration of hypotheses is concerned. 

If the serial model i£ ti^e more nearly correct, this must 
represent a basic limitation of man. It is difficult to think of 
a convincing reason why one should evaluate the .hypotheses serially 
if he is able tq treat them in parallel. 

But even if we assume that one cannot test several hypotheses 
at once, there is still a question about the order in which testing 
is done. One might apply an incoming datum to each of the 
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hypotheses in turn. Alte|:natively , one might focus exclusively on 

one hygothesis untX^ confirming data to accept it, 

or until the evi^deAce agains^ it was sufficient to warrant its 
rQ'jection,^. ift which case atjbention v/ould be shifted to another 
ipossibility . No^ie that in this latter case a datvim cannot be dis- 
carded after being applied to ,the evaluation of one hypothesis 
because it may be germane to the evaluation of. others later, 

- ' One putative advantage of the Bayesian approach (see 
Sectioh 8.6} is that it forces the decision maker to apply an 
iijeoming datujn. to each bf the candidate hypotheses in turn. One 
of th§ implication^ vof this fact is that it minimizes, the need 
, for the decision maker or system to retain data. Assuming that 
the set of hypotheses with which the, decision maker is working 
is complete, and will not be extended, a datum can be discarded 
once it has been assimilated and the probabilities associated 
with all ti^e hypotheses ^revised. 

Siibconscious Processes 

, What is happening at a , subsconscious level is, of course, 
even less Well-understood. The belief has been expressed that 
the brain carries on problem-solving activity' even when one is not 
consciously thinking about a paroblem. Wailas (1926) elaborated 
. and popularized the notion, which he, credits to Helmholtz, that 
creative thinking often involves a period of "incubation," which 
^follows a period of "preparation," and precedes a period of 
"illumination^" During the preparation period, according to this 
view, the problem solver consciously^ labors on the problem,; during 
the illumination period the problem solver becomes aware of the 
solution- for wjiich he was seeking. No conscious attention is 
igiven to the problem during the inciabation period, but, Wallace 
suggests, much subsconecious explpfation of the problem takes place, 

^ While the idea has prima,rily anecdotal , support, the 
testimony of creative thinkers about the way! they have arrived 
at solutions to difficult problems is faixl^^ compelling evidence 
that something of this sort 'does occur. We mention it in this* 
c6htext to make the point that the fact (if it is a "fact) that 
decision makers tend to apply newly acquired data to the ^valuation 
of only one hypothesis at a time, should probably not be taken as 
conclusive evidence' that the credibility of a hypothesis- not under 
consideration has not been affected by those data. Moreover, it 
is at least a plajisible conjecture that the likelihood thdt any 
given hypothesis will"suggest iytself" for explicit consideration 
rtay depend to some degree on suah subconscious activity (Maier, 
1931). ^ " ^ 
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Dreyfus (19 600 has argued that such subconscious, or 
marginally conscious, activity is a general and difficult-to- 
simulate characteristic of man ^s a problem solver. It is this 
ability that makes it possible for him to consider consciously 
only the "interesting" moves- in a game of chess without explicitly 
considering all possible moves and rejecting tliose that are not 
worth pursuing. But subconscious processes are beyond the scope 
of this report, so we will not pursue the topic further, 

8.3 Man A s An I ntu itive I^ g_ici an 

Technically, logic is the discipline which deals with the 
rules of valid inference. The term is used colloquially, however, 
as a synonym for reasoning. It is of some relevance to the 
general problem of decision making, and in particular to the 
problem of training decision makers, to consider whether reasoning 
as it is practiced by people is logical in the technical sense; 
and, to the extent that it is illogical, whether it is illogical 
in consistent ways. A further question of interest is whether 
training in formal logic can reasonably be expected to improve 
decision-making performance. 

Philosophers have not been in agreement on the first 
question. Henle (1962) points out that spme of the 19th century 
writers (e.g., Boole, 1854; Kant, 1885; Mill, 1874) viewed logic 
as the science of the laws of thought, gome more recent writers 
(e g , Cohen, 1944; RuSsell, 1904; Schiller, 1930) have treated 
logic as something quite independent of thought processes and to 
reiect the notion that thinking necessarily conforms to log-ical 
principles.* A middle-of-the-road view is that thinking sometimes 
conforms to logical ^principles— especially when one's explicit 
purpose is to reason carefully and deductively— and sometimes 
does not. . ' ^ 



\ 

*A cynic might assert that few arguments are won or lost on 
logical grounds. Certainly, the'alogical strategems that can 
be applied to arguments are numerous, and perhaps are better 
learned in the course of normal development than are the rules 
of inference. The disputatious reader who feels his arsenal 
of such strategems is deficient is referred to Schopenhauer 
(no date) who provides a veritable cornicopia .pf them. 
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^ Whether or not tliinking is logical may be difficult to 
determine empirically in any particular case, because the steps 
by which one arrives at a conclusion usually are not available 
for observation. As Mill (1874) "points out , since "the premises 
are seldom formally set out,... it is almost always to a certain 
degree optional in what manner the suppressed link shall be filled 
up... [A person] has it almost always in his power to make his 
syllogism good' by introducing a false premise; and hence it is 
scarcely ever possible decidely to affirm that any argument involves 
a bad syllogism" (p, 560; from Henle, 1962). ^ 

Individuals undoubtedly differ greatly in their ability to 
think logically, and any characterization of human strengths and 
weaknesses in this regard is bound to be only partially correct. 
There are many ways in which reasoning can be illogical, however, 
and it is not unreasonable to ask whether some of thr many possibJ^e 
evidences of fallibility are appreciably more common ^.han others • 
Several ways in which human reasoning does seem to depart from the 
ranks of logic have been discussed by Henle (1962). These include: 
failure to distinguish between the^ factual truth of a conclusion 
and the logical validity of the argument on which it is based; 
restatement of a premise or a conclusion, which may have the 
effect of preserving a logically valid, form, v/hile ibhanging the 
substance of the argument; the omission of premises from an argument, 
or the addition of spurious premises. The fallacy of the " mdis- 
tributed middle" is one that has long been recognized as 
particularly bothersome, and involves the assignment of different 
meanings to thfe same term when it appears in different premises. 

Another type of logical error that seem^ to be commonly 
made involves a misunderstanding of tife syllogistic form: "If A 
then B; A; therefore B," or "If A then B; not B; therefore not A." 
These forms may be perve;cted either as "If A then B; not A; 
therefore not B," or "If A then-B; B; therefojre A." Bbth of these 
forms are invalid; nevertheless most. readers will probably recognize 
them as forms that. they have encountered, and perhaps used, ip 
arguments. 

Wason (1974) describes a failure in re'asoning that he has 
observed that seems to be related to this type of misunderstanding. 
Four cards are placed on a table so the subject can see only one 
side of each of them. The cards contain respectively a vowel, a 
consonant, an even number and an odd number. The subject is told 
^that each card has a letter on one, side and a number on the other, 
and is asked which cards would have to be tutned over to determine 
the truth or falsity of the statement: "If a card has a vowel on 
one side, then it has an even number on the other." The majority of 
Wason 's subjects chose either the card showing the vowel and the 
one showing the even number, or just the card showing the vowel. 
The correct answer is: the card that shows the vowel and the one 
that shows the odd number. Only by finding an odd number behind 
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the vowel or a vowel behind the odd number would the statement .be 
falsified. The stud>ents * choice of the card with the even number 
is ^a form of the fallacy known as asserting the consequent: "If 
A then B; B; therefore A.". 

This type of reasoning error occurs wi'th suff icienl^^pnsis- 
tency (at least among college students) to have prompted investi- 
gation by several researchers. A completely satisfactory explana- 
tion has not yet been forthcoming. Wason seems to favor the view 
tha^ the choice of. cards made on an intuitive basis and that 
thfe "reasons" for the choice - which subjects give in response to 
the experimenter's inquiries - are r^allj^ rationalizations • , "This 
'hypothesis is consistent with our crude knowledge * about intuition. 
A verdict may occur to a judge before the grounds which support 
it have been spelled out; a chess player may "see" a good move^ and 
then analyze the continuations which validate it. Such thought ^ 
suggests a processing mechanism which operates at different levels" 
(p*385). 

The last chapter on the topic of the relationship between 
logic and thought has not been written. And it cannot be until ''^ 
much more is known ^out the workings of the human mind. The ^ 
immediate challenge for training research is to identify ways to 
improve the capability of individuals i!o reason logically, or at 
^east'to recognize and be able to avoi^d the more common illogical 
pitfalls. 

^ • ^ Man as an lntuiti\^^ ^^j:^^ ^ t^.^ ian 

It is* quite clear that most individuals could 'Manage to get 
through life without e /er explicitly assigning a numerical 
probability to an event. Undoubtedly, the vast majoJ^^Y of people 
do so. It seems safe to assume, however, that people do make 
judgments of likelihoods, and that these judgments — even though 
nonnumeric, and often implicit — condition their behavior. An 
individual carries ah umbrella because he thinks there is a good 
chance of rain, or buys stock that he expects to appreciate. One 
purchases . life insurance^ before boarding an airplane because one, 
in effect, has considered the likelihood that the plane will go 
down during that^flight to be nonnegligible; the fact that he 
boards the plane at all is probably evidence that he also considers 
that likelihood to be something less than certainty. One chooses 
one among three job opportunities, because the chances of success 
and advancement are perceived as greater in the case of the selected 
job than in that of the others. In short, although most of us do 
not attempt to assign numeric probabilities to possible situations 
or events, we behave as though our choices had been dictated by 
reasoning of the sort: this event is more likely than that, or the 
likelihood of this situation is great enough so that I had better 
do thus and so in order to be prepared if it should occur. 
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A question of some practical interest, therefore, is that of 
how effectively such judgments are made. For many situations, 
there is no way to answer this question objec-tively. The individ- 
ual who selects one job from among 'three possibilities because he 
considers the likelihood of success to be highest for that case 
will never know for certain whether his judgment was correct. - 
There are also, however, many^ situations for which the "objective" 
probabilities of events are known or can be determined, and we 
can at least ask how well people do when Asked to estimate 
probabilities explicitly in thdse cases. The literature that is 
relevant to this question falls fairly naturally into three cate- 
gories. First are the studies that deal with people's ability to 
estimate the statistical properties of samples that they are 
permitted to observe. Such studies concern relative frequencies 
rather than probabilities, but to the degree that our ideas about 
probabilities are based on, or influenced by, percei\ frequencies 
they are germane. Second are some studies that have to do with 
the extent to which ^people's intuitive notions about the probabil- 
ities of events correspond to, or conflict with, the implications 
of the theory of probability as represented in the probability cal- 
culus. Third are numerous recent experiments that consider the 
specific question of how effectively people function as Bayesian 
data aggregators. In this section we will consider briefly the 
first of these three categories of studies; in Sections 8.4 and 8.5 
we will consider the last two. 

People appear to be reasonably good at perceiving proportions, 
or the relative frequencies of occurrence, of both sequential and 
simultaneous events (Attheave, 1953; Peterson & Beach, 19.6 7; 
Schrenk & Kanarick, 1967; Erlich, 1964; Vlek, 1970) and at esti- 
mating the means of number sequences (Be'ach & Swensson, 1966; 
Edwards, 1967)., Inferences ' concerning the median or mode of a 
skewed distribution (assuming the subject knows the definitions 
of these terms) are fairly accurate, and the estimated mean of such 
distributions tends tb be biased in the direction of the median 
(Peterson & Beach, 1967). One's confidence is one's estimate of 
the mean or the variance of a population appears to increase as the 
sample size increases (Peterson & Beach, 1967; but see also Pitz, 
(1967). 

Estimates of the variability of a set of data often tend to 
decrease as the mean increases (Hofs tatter, 1939; Lathrop, 196"7; 
Peterson & Beach. 1967) . Peterson and Beach (1967) point out that 
while the notion that variability is necessarily inversely related 
to the mean is erroneous, it is intuitively compelling. "Think 
of the top of a forest. The tree tops seem to form a fairly smooth 
surface, considering that the tree may be 60 or 70 feet tall. Now, 
look at your desk top. In all probability it is littered with many 
objects and if a cloth were thrown over it the surface would seem 
very bumpy and variable. The forest top far more variable than ' 
the surface of your desk, but not relative to the sizes of the 
objects being considered" (p. 31). One is led to wonder whether 
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. the finding that estimated variability tends to decrease with 
increasing "mean might be due in part' to failure by the subject 
to ^understand that it is/ an estimate of absolute variability that 
he is to produce'. Relativ e variability probably often does de- 
crease as the mean increases (to cite Peterson and Beach's tree 
top example) , and without explicit instructions to the contrary it 
would not be unreasonable for a subject to suit the terms to the 
context, as one does when one speaks both of a small skyscraper 
and a large dog; 

8 • 5 Intuitive Probab ility Theory 

How closely do man's intuitions about probabilities corre- 
spond to the implications of probability theory? The question 
cannot be answered decisively, but a number of pertinent observa- 
tions can be made. For example, people often seem to find it 
difficult to believe that the outcome of an event can 
independent of what has preceded it. This difficulty is bometimes 
manifested in the "gambler's fallacy"- (a fallacy, that competent 
gamblers probably would not make) , one form of which holds that a 
run of successes increases the likelihood of a failure, or vice 
versa (Cohen & Hansel, 1956). Another example of assumed dependence 
among successive events has been noted by Jarvik (1951), who found 
that when given a two-alternative prediction task, subjects often 
tended to predict the more frequent event after one occurrence of 
the less frequent event and to predict the less frequent after two 
consecutive occurrences of the more frequent event. 

Several experimenters have found that man does not estimate 
the probability of compound events very accurately. In particular 
when assessing the likelihood of the joint occurrence of several 
independent events, he tends to produce estimates that are too 
high (Cohen, Chesnick, & Haran, 1972; Fleming, 1970; Slovic, 1969). 
Conversely, when estimating the probability of disjunctive events-- 

several specified events will occur- 
^ K^^^n^ 'Wf^'^S'"''^ estimates that are too low (Cohen, Chesnick, 
^ho ^JoK Tversky & Hahneman, 1974). The overestimation of 

the probability of conjunctive events is consistent with the ob- 
servation that people frequently base judgments of the degree of 
oJ^'fni^^o^? between two events on those cases in which the outcomes 
of interest d6, occur together without giving sufficient considera- 
tion to those cases m which they do not (Peterson & Beach, 1967) . 

What is of more interest than the fact that man's intuitions 
sometimes lead to incorrect judgments about event probabilities is 
the question of the extent to which the failings of' intuition — 
at least insofar as they are systematic--are explainable in terms 
of identifiable ways in which such judgments are made. In a 
recent series of studies, Tversky and Kahneman (1971,. 1973, 1974; 
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Kahnoman & Tversky, 1972, 1973) 'have explored this question4 The \ 
general approach in these studies and in those of others whosjiave 
conducted similar investigations (e.g., Alberoni, X962; Tan6",^1964; 
Wagenaar, 1970) has been to ask people to estimate the probability 
of the occurrence of a hypothetical event, or, perhaps more commonly^ 
to indicate which of two such events is tlie more probable. One 
might be asked, for example, to indicate which of the two following 
sequences of coin tosses is the more likely, HHHHTTTT or HHTHTTHT; 
or to indicate which of two hospitals — which record approximately 
15 and 45 births a day, respectively — would have the largest 
frequency of days on which more than 60% of the babies born are 
boys • 

The results of these studies have revealed a number of ways 
in which th^. answers that people give to such questions depart 
systematically from the objective probabilities of tl events as 
inferred from the application of probability mathematics. Tversky 
and Kahneman attribute such failures in judment to the heuristic 
principles that people often use when attempting to* estimate 
pi'ob abilities or relative likelihoods. 

It will be helpful, before considering some of Tversky and 
Kahneman 's specific results to digress briefly to consider the 
notion of a, heuristic principle or procedure. The term "heuristic," 
which comes from the dreek heurisk in/ meanin^f "serving to dis-, 
cover," appears sporadically in the literature of philosophy and 
logic a^ the name of a branch of study dealing with the methods 
of inductive reasoning. It was revived by Polya (1957) in his 
classic treatise on problem solving, and used to connote inductive 
and' analogical reaspning leading to plausible conclusions, as 
opposed to the deductive developments of rigorous proofs. In 
recent years, computer scientists, and especially researchers in 
the area of machine intelligence, have appropriated the term to 
connote "a rule of thumb, Strategy, trick, simplification, or other 
kind of device which drastically limits search for solutions in 
large problem spaces" (Feigenbaum & Feldman, 1963, p. 6). In short, 
a heuristic principle or procedure, usually referred to simply as a 
heuristic, is a means of making an inherently difficult problem more 
tractable. The criterion by which a heuristic is measured is its 
usefulness. It is important to bear in mind, however, that 
heuris'-.ics are not expected to lead invariably to coi^rect solutions. 
"A 'heuristic program',' to be considered successful, must work well 
on a variety of problems, and may often be excused if it fails on 
some" (Minsky, 1963, p. 408). 
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8.5.1 Representativeness 

TverskW and Kahneman describe two heuristic principles-^- 
. representativeness and' availability—which they feel account for 
many of- the $ysteinatic judmental biases that they and other 
investigators have o^^served. According to the representativeness 
principle/ "the subjective probability of an events or a sample, 
is determined by the^ degree to v/hich it: (i) is similar in 
essential .characteristics to \j.ts parent population; and (ii) 
reflects the salient features of the process by which it i-s gen- 
erated" (Kahneman & Tve;rsky, 1972, p. 430). Several examples of 
the application of this principle are given; two will suffice for 
our purposes, one illustrating each of the ^ubprinciples. 

, The importance of the similarity between the judged event 
and the parent population is illustr.ated by the follow: "^g question 
"All families of six children in a city were surveyed. In 72 
families* the exact order of births of boys and girls was GBGBBG. 
What is your estimate of the number of families surveyed in which 
the exact order of births was BGBBBB?" (Kahneman & Tversky, 1972, 
p. 432) . If the probabilities of male and female births were 
exactly equal, .the two birth sequences would be equally probable. 
(Apparently, the frequency of male births is slightly higher than 
that of female births, so the lattef s,equence is slightly more 
probable than the former.) About 80% of the sx±)jects (high-school 
^students) who were asked this question judged the latter sequence 
to be less likely than the former; the median estimated number of 
families with this birth order was 30. Kahneman and,Tversky 
attributed this result to the fact that the two birth sequences, 
while about equally likely, are not equally ^representative of 
families ij^ the population. The former sequence is^ more similar 
to a larger proportion of the population, both in terms of the v 
relative number of girls and boys, and in terms of the length of 
runs of births of the same sex* 

The second way in which the representativeness heuristic 
manifests itself — in sensitivity to the degree to which an event 
reflects the salient features of the process that generated it — 
is illustrated by the tendency of people to consider regularities 
in small samples to be inconsistent with the assumption that such 
samples were generated by a random' process. Thus, when people are 
asked to produce random sequences such as the results of an imagin 
series of coin tosses, they tend to produce fewer long runs than 
would a truly random process. Moreover, in judging the randomness« 
of small samples, they are, likely to reject as nonrandom many of 
the samples that a random process does generate. Kahneman and ^ 
Tver sky .characterize the intuition that produces such judgmental 
biases as a belief that a representative sample should represent 
the essential characteristics of the parent population, not only 
globally, but locally in each of its parts. In other words the 



NAVTRZUEQUIPCEN 73-C-0128-1 

o 



observed behavior is consistent with the belief. that the law of 
large numbers applies to small numbers as well (Tversky & Kahneman, 
1971). . • . ' 

The application of this heuristic could lead one to the sort 
of fallacious thin^'inq illustrated by the conclusion that the ^ 
probability of finumg more than 600 boys in a random sample of 1000 
children is the same as that of finding .more than 60 boys in a random 
sample of 100 children. The probability of the latter event is, 
bf course, much greater than that of tWe former, Kahneman and 
Tversky .(1972) showed that people (at least high-school students) 
do virtually ignore the effect of sample size when estimating the 
probabilities' of random^eVents of this sort. In general, the ' 
estimates made by Kahneman and Tversky 's subjects, when asked to 
judge the probability of events that have a binomial distribution, , 
were much mqre appropriate for small samples (e.g., ^) than for 
large samples (e.g. , lOOror IflOO) . In other words, foi large s^amples, 
subjects tended 1^6^ tinder estimate grossly the probability of high- 
probability events ^nd overestimate the prpbability of low-proba- 
bility events, and the magnitude of the miss increased with the 
size of the sample. 

8.5J2 Availability 

The availability principle, 'according to Tversky and Kahneman 
(1973) is used whenever one .bases estimates o^f frequency or prob- ^ 
ability on .the ease with which instances or associations are ^ 
called to mind. For example, when asked to estimate the relative 
likelihoods of heart attacks for men and women, one might think of 
male and female victims of heart , attack among one's personal acquain- 
tances and- take the ratio as. an estitnate of the relative likelihoods 
in the population. Or, if asked to judge which of two letters occurs 
the more frequently as the first letter of English words , one might 
attempt to think of a few \ft^ords of each class and make the judgment 
on the basis of the rapidity with which examples come to mind. 

Tversky and Kahneman point out that "availability" is an , 
ecologically valid cue for the judgment of frequency because # in 
general,* more frequent events are easier to recall or imagine than 
infrequent ones. However, availability is also affected' By various 
factors/ v;hich are unrelated to actual frequency. 1^ the availability 
heuristic is applied, then such factors, will affect the perceived 
frequency of classes and the subjective probability of events. 
Consequently- the use of *the availability heuristic leads to 
systematic biases" (1973, p. 209). 

As one example of how application of the availability heuristic 
can lead to an erroneous judgment, Tversky and Kahneman report the 
following experiment. Subjects were asked to estimate the number of 
different remember committees that can be formed from a group of 10 
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people. The estimates tended to decrease with increasing r for 
.values of r between 2 and 8. In particular, subjects typically 
Dudged It to be possible to form mahy more committees of size, 2 
' IaIIs fJ?!-?;.'"^^'' in fact the same number is, possible in both- 
cases.. (Similar results were obtained when subjects were asked 

* i^ntrS!^ ""S^^^ °- 'iifferent^ patterns of r stops that a bus 
could make while traversing a route with 10 stations between start 

A l^'u ' The explanation for this result, according to Tverskv 
n^l S^^^'f ^^''^ ^^^^ committees {Of two members are ^ 
more readily imagined than those of eight, and, ^consequently, appear 
to be more numerous . ^ at-t"=a-'- 

The major difference between the heuristic principles of ~" 
representativeness and availability, Kahneman and Tversky suggest, 
^K-t^.^^^ >^?ture of the judgments on which the subjective prob- 
ability estimates are based. "According to the representativeness ^ 
heuristic, one evaluates subjective probability by the' degree of 
correspondence between the sample and the population, or between 

^ "^^^^ heuristic, therefore, emphasizes 

the generic features, or the connotation, of the event. According 
to the availability heuristic, on the other hand,' subjective 
probability is evaluated by the difficulty of retrieval and -on- 
f^^^^^o °^ instances-: It focuses, therefore, on, the particular 
instances, or the denotation, of the event. Thus, the represen- 
tativeness heuristic is more likely to be employed when events are 

u'''' ^^"""^^ °^ ^^^^^ general properties; whereas, the 
availability heuristic is more likely to he employed when events 
are more naturally thought of in terms of specific occurrences" 
(Kahneman &. Tversky, 1972,, p. 452). A feature commpn to both 
heuristics is their reliance on mental effort as an indicant of 
nn^E^?^-""^ probability. "It is certainly harder to- imagine an 
uncertain prcJcess yielding a nonrepresentative outcome than to 
imagine the same process yielding a highly representative outcome, 
bimilarly, the .less available the instances of an event, the harder 
It X.S to retrieve and construct them" (ibid, p. 452). 

8.5.3 A Methodological Consideration " 

There is a methodological consideration relating to 
some of the findings of judgmental biases that deserves more 
attention than it has received. This has to do with the possible 
I?, lu ^f?5uage ambiguities. We have already alluded more than once 
to the well known fact that the meaning of language is conditioned 
by the situation m which it occurs. * To borrow an example from 
^^f^^^f y phrase like 'stay near me' can mean anything from 

vShL J° ^^^^^ "'^l^ ^^^y'' depending upon 

l^n?^^f.i\i: ^'^'^^r^^'^ J° ^ ^hil'^ in-a- crowd or a fellow astronaut 
Sf^ihP^L,nL"'?rI ^2^' Although it seems unlikely that many 
of the results that have been mentioned above can be attributed to 
the imprecision of language, the possibility that some of them 
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may be based, at least in part, on this factor should not be 
overlooked. The finding .that the estimated variability of a set 
of -data tends to decrease as the mean increases was mentioned in 
a preceding section as one possible case in point. Tversky and _ 
Kahneman's* finding that people judge it to be possible to form a - 
larger number of different 2-man . committees than 8-man committees 
-from a pool of 10 men may be another. There is a way of defining 
"different" (e.g., "having no people in common ) such that tne 
judgment would be valid, and before one can take the results as 
evidence of faulty intuitions concerning- combinatorics , one must 
be certain that none of the "^subjects is using such a definition. 
Our guess is that language ambiguities will not go far toward 
explaining the results obtained by Tversky and Kahneman, but it 
seems conceivable that they may have played some role, and some 
further research. might be directed toward determining the extent 
of that role. 

8.5.4. Training and Intuitive PfSbability Theory 

We have reviewed these results at some length because ^^is 
general line of research strikes us as being not only exceptionally 
interesting from a theoretical point of view, but of considerable 
practical significance. To the extent that the heuristics that ^ 
have been identified are representative of the ways m which people 
generally make judgments of likelihood, it is clearly .important to 
determine those conditions under which they lead to erroneous 
judgments and those under which they do not. Tversky and' Kahneman 
have demonstrated that there are at least some situations in which 
judgments, that are presumably based on identifiable heuristics, 
err in systematic ways. This )does not, of course, establish that 
these heuristics are, on balande, bad, as they are careful to 
point out. What one would like to know is the relative frequency 
*with which they lead to erroneous . decisions m practical real-life 
situations. From the point of view of the training of decision 
makers the question is how to foster the use of such heuristics m 
situations in which they are most likely to be effective, while 
discouraging their use in situations in which they are likely to 
lead to erroneous judgments. Perhaps at least a small step m 
that direction would be to make decision makers explicitly aware 
of the nature of 'the heuristics that tend to be used m estimating 
probabilities, and of the types of- erroneous decisions to which 
they can somejiimes lead. 
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" Bayes i an Inference 

Undoubtedly the most widely advocated formal approach to the 
application of incoming data to the evaluation of hypotheses is 
the "Bayesian" approach. Because it has attracted so much atten- 
tion and has been the focus of so much research, we will consider 
It m somb detail. 

8.6.1 Bayes Rule 

It is necessary to begin with a set of mutually exclusive 
and exhaustive hypotheses, H^, concerning the state of the world. 
To each of these hypotheses one must assign. a probability, p(H-), 
that that hypothesis is true. Because these hypotheses are, by 
definition, mutually exclusive and exhaustive, it follows that the 
a priori^ probabilities sum to one, r?e., 

•?P««i)=l. (1) 

Q Inasmuch as the hypotheses that one is considering are likely 
o have different .implications concerning what might be observed 
under specified conditions, it seems intuitively reasonable that 
one should be able to increase one's degree of certainty concern- 
ing the truth or falsity of any given hypothesis by making appro- ' 
priate observations. For example, if h. implies D, and if D is 
observed, then the credibility of might reasonably be expected 
to be increased. (The truth of H. is not proved by such an obser- 
vation, of course, inasmuch as it does not follow from the fact 
that implies D that D implies H.; as was pointed out in Sec- 
tion 8.3, inferring the truth of Hi from the observation of D 
would involve the logical fallacy known as " asserting the con- 
sequent.") If both H. and H. could lead to D, but the likelihood 
of D given H- is greater thafi its likelihood given H., then our 
intuitive notions about evidence suggest that the obServation of 
D should increase our confidence in H. somethat more than our 
confidence in H^c These notions were expressed formally by the 
18th Century British minister, Thoma^ Bayes, in the so-called 
"inverse probability theorem" ~a theorem or rule that has been 
the subject of much debate. 

Bayes rule expresses p{H.|D), the probability that H. is true 
given the observation, or datum, D, as a function of p(d|h.), the 
probability that D will be observed given is true, and ^(H,-)/ 
the probability that Hi is true as determined prior to the obser- 
■vation of D. The probability of an observation given a hypothesis, 
p{D|H) is usually referred to a*s a conditional probability; the 
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probability of a hypothesis given an observation, p{h|p)., is usually 
called. a posterior pr6bability. Bayes rule defines a procedure for 
using the fact that D has been observed, to adjust one's estimate of 
the probability that Hi is true. The rule may be written as 



p{D|H^) p(-H^) 

P{Hi|D) = : , (2) 

E pj(D|H.) p(H.) 

i=i , ^ ^ . ■ V 

I 

I 

Where n is the total number of hypotheses in the set. Because 
Zp(D|H.) p(H.) = P{D), equation (2) may be simplified ,tO 
j P D 

p{D|H.) p{H.) 

P(H. Id) = i ^ . (3) 

p{D) 

When a sequence of observations is made, the rule is applied 
recursively, "and the value of p{Hi|D) that is computed as the 
result of one observation becomes the pCH^) for the following 
computation. That is to say, the posterior probabilities result- 
ing from one observation become the prior probabilities for the 
next one. Thus, equation C3) may be written more appropriately as; 



Pn^Hil^) = ^ , (4) 

where PnCHilD) represents p{Hi|D) after the n^^ observation, and 
PQ{Hi|D7 or, more appropriately, PqCH-j^)/ is understood to be the 
probability of Hi before any observations are made. We will follow 
the convention of using subscripts only when they are essential 
for clarity. 

Bayes rule states, in effect, that if the prior probability 
of a hypothesis being true, p{Hj^) and the probability of observing 
a particular datum given that hypothesis is true, p{D|Hj^) are known 
for all i, then the probability that the hypothesis is true given 
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tfiat the datum has been observed, p{Hi[D), can be calculated in 
a straight£orv"ard way, .In many decision situations, p(Hj^) and 
p(D|Hj^) are not known, and cannot be determined objectively; 
therefore, they must be estimated. The significance of the rule 
stems from the assumption, for which thelre is some evidence that 
will be considered later, that people are better at estimating 
conditional probabilities, p(DjH), than at estimating posterior 
probabilities f p(h|d). Obviously, if they w^ere invariably very 
good at estimating p(H]D) there would be no need to make use of 
Bayes rule to calculate this value; it would suffice to have the 
decision maker estimate it directly. 

• ' ' -I 

8.6.2 Likelihood^ R^ltio 



In order to %ake use of Bayjes rule it is not necessary to 
require that an individual estimate probabilities expl'citly. An 
alternative procedure is to have him judge the ratios of pairs of 
conditional probabilities. Such ratios are referred to as 
likelihood ratios. The likelihood ratio of D given relative 
to D given H2 may be expressed as follows: 



The . attractiveness of likelihood ratio stems from the fact 
that people often find it easier to make the implied judgment 
than to estimate conditional probabilities directly. The type of 
judgment that is required in this case is of the sort "Event D 
is X times as likely if is true than if true." Neither 

of the conditional probabilities need be specified explicitly. 
A disadvantage associated with its use is the fact that a great 
many more judgments are required with respect to each observation. 

8.6.3 Other Methpds for Obtaining Probability Estimates ^ 

Other methods have been used to obtain -probability estimates 
without having the subject explicitly produce numerical values. 
Fqr chips-in-urn problems, fojr example, Peterson and Phillips 
(1966) have had sulDjects adjust markers on a scaled 0-to-l con- 
tinuum so that each interval is equally likely to contain the 
true proportion of chips of a specified color. Organist (1964) 
developed a simple answer chart which forced a' subject to make 
his distribution of probabilities over the possible hypotheses 
sum to one and also specified what his payoff would be for each 
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hypothesis if it, were* correct, given the«?probability that he 

attached to it. Shufort^ C1967) describes a computer-controlled 

system which presents a subject with a set of hypotheses ana 

allows him to specify probabilities by adjusting thfe lengths of 

lines associated with the hypotheses by pointing at them with a \ 

light pan. When one line is lengthened or shortened, compensatory 

.adjustments are automatically made in the remaining line^ so that 

the probabilities always sum to one. This system also pirWides the 

user with information coijcerning the implications cX>f his probability 

assignments vis-a-vis his payoff, given the truth of <any 

parti cu 1 a r hy pottlSSi *^'*^^*'**^*^*^*^'*^*'***"^****'^"***'***^^^ -mi***** f«.«««w««««»»*«jufl*5«^M**»i««i«Mi*^ 



Intuition suggests that the more disparate the implic'^tions 
of two hypotheses, the more informative data should bs concerning 
which of the hypotheses is likely to be true. In a Bay^sian 
context the infonnativeness, or "4iagnosticity, " of data is defin^d^ 
in terms of the likelihood rati6. Specifically,, the magnitude of 
a likelihood fatio is said to represent the diagnosticity^.of a 
datum with respect to the two particular hypotheses involved. The 
more the ratio differs from 1:1, ^in either direction,^ the more 
informative the datum is with ^respect to which of the, hypotheses 
under consideration is correct, and the more the distribution of 
probabilities over' these hypotheses will chapge as a consequence., 

8.6.5 *.Odds . ^ • ' ^ 

The ratio of two posterior probabilities is referred to as 
the posterior *^odds" with respect to the associated hypotheses. 
The posterior oddh of H-^ with respect to H2 may be expressed as 



P„(Hi|D) 



PnCHjiD) _ ' p.(D) , 




p{D) 



(6) 



or, equivalently, as 



Pn-l<"l|P^^ 

Pn(H2l°) p{D|H2) Pn-l^Hpljl) 



p^{H^iD) _ p{D|H^f 



(7)- 
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which is to say that the. posterior odds is,, simply the prior odds 
multiplied by the likelihood ratio. Letting fin;i j represent the 
odds of Hi with respect to after J±e n^h observation, we may 
express the relationship as follows;"^ 



. . = L. . ft ^ / ' 



Obviously 



and 



3/1 i/j 



Often it is clear from the context which of the two terms of 
either an odds ratio or a likelihood ratio is to be the numerator 
and whiqh the denominator, so the subscripts are omitted and the 
expression is written more simply as ^ 

"n = ^Vl- fli) 



It is essential, however, that the same hypothesis, whether H. or H . , 
be represented in the same position (numerator or denominator)* in 
both ratios 



8.6*6 Applications of Bayes rule in The Two-Hypothesis Case. 

To summarize jvhat has been said so far, Bayes rule represents 
a procedure, for evaluating hypotheses in situations that have the 
following characteristics: (a) the possible states of the world 
can be explicitly represented by an exhaustive, and mutually 
exclusive set of possibilities; (b) discrete observations may be 
made in an effort to find more infomation about the actual state 
of the world; and (c) for the data obtained from each observation, 
it must be reasonable to assign a number, that represents the 
probability that those data would have been obtained, given the 
truth of any specific one of the hypothesized states of the world. 
In order to get an appreciation of; how Bayes rule extracts 
information from data, it will be helpful to consider some concrete 
examples of decision tasks to which the rule might be applied. We^ 
will focus first on the simple case in which the hypothesis set 
contains only two alternatives. 
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Imagine an urn containing! red and black chips. Suppose two 
hypotheses, and Hj/ are stated, one and only one of which i's 
truer concerning what proportion of the chips in t4ie urn are red. 
The task is to decide which of these hypotheses is the true one.^ 
Data are obtained by sampling chips one at a time, replacing each 
chip after it is examined. Assume that the chips are thorough^ly 
mixed jDefore each observation and that the probability of drawing 
a red chip on* a trial is exactly R/R+B, where R is the number of 
red chips, and B the number of black chips, in the urn. 

Suppose the first hypothesis, , is that 70% of> the chips 
are red^^ and that the second hypothesis, H^, is that 20% of the 
chips are red. Suppose further that the prior probabilities are 
equal, that is, Pq(H^) = Po^"2^ .5.^, Figure 1 shows how , 
p(H^|D) and p(H2|D) change as a result of applying Bayes rule tq 

the data obtained in the following ten successive observations: 
RRBBRRBRRR. Figures 2 and 3 show the odds, and the uncertainty, 
in the information theoretic sense of the word, change from ob- 
servation to observation. Uncertainty is, of course, a monotone 
but nonlinear function of the difference between the prob.abilities 
associated with the two hypotheses. . ^ ' . • 

■ ''■\ ) 

Note that the effect ofvap observation is not/ i>ecessarily to 
decrease the amount of uncertainty concerning wh:^h hypothesis is 
true. If the distribution of Pn(H.) favors the incorrect hypothesis, 
uncertainty is very likely to increase as a result of observing .^ 
data before it decreases. Even if the distribution of P^{H.) 
favors the correct hypothesis, or weights both hypotheses equally, 
uncertainty may» increase on individual trials. In this c^se, how- 
ever, it will decrease on the average , assuming unbiased sampling. 

Another interesting and perhaps counterintuitive observation 
concerning figure 1 is the very large effect that the one or two 
initial observations can have . In our example, the initial 
drawing of two successive reds had the result of making one of the 
(initially equally likely) hypotheses over twelve times more likely 
than the other. 

Intuitively, one would expect that the degree of confidence 
that one should have that the proportion of reds and blacks in one's 
sample reflects the true proportion in the population should depend 
on the sample sie.e. That the application of Bayes theorem does not 
violate this intuition may be seen by comparing the probability 
distribution after the third observation and after the sixtfh obser- 
vation (figure 1). In both cases, red, chips have comprised 67 percent 
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PERCENT OF REDS IN SAMPLE 

H^. 70% Red; n^: 20% Red ; 
Po(Hj^) = Pq(H2) = .5 

Figure 1. Changes in posterior probabilities, 
P(Hj^|D) and p(\l^\D) as a result of 
the indicated observations of Red and 
Black chips 
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OBSERVATION NUMBER 

RRBBRRBRRR 

DATUM 

100 100 - 67 50 60 67 59 62 67 70 
PERCENT OF REDS IN SAMPLE 



H^: 70% 20% Red; 

PqCHj^) = Po(H2) = .5 

Figure 2. Changes in odds, ilj^ 2 ^ result of the 

indicated observations of Red and Black chips 
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3 4 5-6 7 8 
OBSERVATION NUMBER 



R B 



B 



R R 
DATUM 



B R 



9 
R 



100 100 67 50 60 67 59 62-- 67 
PERCENT OF REDS IN SAMPLE 



10 .11 
R 
70 



H^: 70% Red; E^i 20% Red; 
P0%^ = Po^«2) = -5 



Figure 3. Changes in uncertainty concerning hypotheses 
as a result of the indicated observations of 
Red and Black chips 
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of the sample; however, the uncertainty is less following th^sixth 
observation than following the third, reflecting the fact that the 
sample size Was larger in the former case. 

Figure 4 shows hov7 the probabilities change over the course 
of ten observations in ^which reds and blacks occur with the same 
frequency but in a different order. In particular, the first two 
observations in this case produced blacks, and the second two, reds. 
Observations 5 through 10 are assumed to be the same as m the 
original example. Note that by the end of the fourth trial, the 
proportion of red and black draws was the same in. both examples; 
consequently, thg probability distributions are the same at this 
point and thereafter. This illustrates the fact that the Bayesian 
calculation of p(H.|D) is pa^h-independent , in the sense that the 
effect of an observation is strictly dependent on the current value 
of'p(H.), and independent of the particular sequence of obs,ervations 
on whiih that value is based. The calculation is also andependent 
"of the number of observations on which the current value of p(H^) 
is basedT Note that, this point is different from the one made above 
concerning the effect of sample size ,on uncertainty. The point 
that was made above was that the probability that a given proportion 
of reds in '.one's sample accurately reflects the proportion m the 
population -increases with sample size. ^The noint here is that the 
effect -that ^n observation will have is independent of how 
p(Hj^) got to be whatever- it is- * ^ 

Figures 5, 6 and 7 illustrate' the effects of setting 
the initial values of p(H, ) and piU^. to something other than .5. 
The sequence jDf draws is identical to that in- figure 1, and con'- 
sistent with what might be expected if the true hypothesis were H^^. 
In each figure, one curve shows the effect of 'these observations 
given that Pn (H, ) = .8; another shows the effect given that Pq^Hj^) 
=' .2, and thg tMrd.-^ represents Po(H ) = .5. The ma?.ji thing td 
'notice is that the effect of an initial incorreet bias is larg^^ly 
nulled out by relatively few observations. This point is frequently 
made by proponents of Bayesian information7processing systems m 
response to the observation that a ^priori probabilities are some-- 
times difficult to assign on anything other than an arbitrary basis. 
A fact that usually i^s not pointed out is illustrated in figure 7: 
changing the distribution of a priori probabilities shifts thr 
function relating log odds to data by a constant. 

8.6.7 Expected Effects of Observations on Hypotheses 

- ^ 

In the foregoing examples of the application of Bayas rule ,^ we 
have considered how probabilities may change as a result of a 
sequence of specific observations. ' It has been apparent from these 
examples that the effect of an observation sometimes is to increase 
the probability associated with the true hypothesis and sometimes to 
decrease it. On the average , however, we expect the probabili^ty 
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3 4 5 6 7 8 
OBSERVATION NUMBER 



B B 



R R R R 
DATUM 



B 



R 



9 10 11 
R R 



^ 0 25 50 60 67 59 62 67 ' 70 
PERCENT OF REDS IN SAMPLE 



H^: 70% Red; U^z 20% Kea; 
PqCH^) = p^(u^) = .5 
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Figure 4. Changes in posterior probabilities, p(Hj^|D) 

and p{H^|D) as- a result of the indicated 

observations 'of Red and Black chips 

(Note that the results of the observations 
are the same as in figure 1 except that the 
• first four produce a different ordering of 
Reds and Blacks.) 
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4 S 6 7 
OBSERVATION NUMBER- 



R R B B R R B R R R 
. y DATUfl^ 

100 103 67 50 60 67 59 62 67 70 
^ PERCENT OF REDS IN SAMPLE 



Hj^: 70% Red; H2:.20% Red 



Figure 5. Effects of indicated observations on . 

p(H, (d) for different values of Po(H,) 
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3 4 5 6 7 8 
OBSERVATION NUMBER 

<*• 

B B R R 
DATUM 



R R 



67 Bd 6Sd 67 62 67 

PERCENT OF RED3 IN SAMPLE 



10 

R 

70 



11 



H^: 70% Red,; 20% Red 



Figure 5. Effects of indicated observations on odds, 
flj^ 2 different initial values of p(Hj^) 
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O R (H,)-.80 



(H>)«.20 




3 • 4 5 6 7 8 
QBSERVATIW NmBER 



R 



B B 



R R 
DATUM 



B R 



100 100 67 50 60 67 B9 62 67 
PERCEhfT OF REDS, IN SAfPLE 



10 11 
R 

70 » 



H^: 70%, Red; U^: 20% Red 



Figure 7. Effects of indicated observations on 

'uncertainty f^ different initial valjes 
of p(Hj^) 
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associated with the true hypothesis to increase with each observa- 
tion, and that associated with a false hypothesis to decrease, as- 
suming an unbiased sampling of the' data. We turn now to a con- 
, sideration of how p(Hi|D) can be expected to change, on the average, 
as a result of applying Bayes rule, if chips are drawn from an urn 
that contains reds and blacks in the proportion indicated by a 
specified hypothesis, say H.. 

It will facilitate the discission to begin by considering ^11 
possible outcomes of a specific experiment. Figure 8 shows all 
possible effects of four observations on p{Hi!D), in the case of 
\ our example of H f 70% k, H : 20% R, and pCH^) = .5. Each node 
yin the graph represents one possible value of p{HiiD) after th^ 
number of observations indicated on the abscissa; no values are 
possible other than those represented by nodes, (By rotating the 
graph m figures about a horizontal axis passing tftr-ough the .5 
point* on the ordinate, one would produce the graph of p^H^lD); 
which is to say, each of the points in the qraph of plB-^W) i^ the 
complement of appoint in the graph of pCHjlD)-) In general,- after 
N Observations, pCH^lD) will have one of M+1 ^possible values, kites 
tvo) observations, for example, p{Uy\D) will hav'e one of the three values 

.bbB, or .123. The number above each node indicates the 
number of ways to arrive at that node. There are three ways, for 
example-, to arrive at the node at p(H,!d) = .821, N ^ 3: RKB, RBR 
and BRR. The set of numbers associated v/ith a given value of N 
. ^^^] be recpgnized as the coefficients of the terms of the expansion 
of (a+b) , the so-called "binomial coefficients." in our appli- " 
cation, each of these cotef f ici^nts, which may be written as (N), 
represents the number of ways that N events can be composed. 
. of fti events of one type and N-m. of another. The events of interest 
in our case are draws of chips from an urn, and fjie two types' 
are draws "of red and black chips, respectively. The sum of these 
coefficients for given N, - - , 

m=0 , •/ • ^- ' ^ ^ 

-represents the nui^ber of uniquely ordered sequences of reds and ^ ' -» 
blacks tfiat'can result from N^aws. Inasmuch as the effect of 
applying Bayes rule to a sequence%f data is insensitive to -the 
} order in which the data are considered, ih is convenient to th'ink 

of all sequences having the sam^ c^bination of reds and blacks. 
. as the same outcome, irrespective of the order in which .the reds- 
and, blacks have occurred. Thusf'the effective number of possible ^ 
/t, outcomes of N -draws is N+1 rather j^a^ 2^. L y'^^ 

■ Figure $ shows the graph oT^ossible ot^mes for our ' 
hypothetical 'experiment as they pe?Fain~Eb p(H£"|D). By the ^ 
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algebra of expectation, the expected , or mean value of p(PL |D) 
after N draws is the weighted sum of all^possible values, e^ch 
value being weighted by its probability of occurrence, Jn terms 
q£ figure 8, the expected value of p(Hi|b) after N draws may'be 
found by multiplying the product of the value of each node above. 
N on the abscissa by tjhe probability ^qf arriving at that node, 
and summing over the^fee products. ^ ^ 

■'I 

Suppose that the probability tha.t an observation will yield 
a red chip (solid lines) is q, and the probability that, it will 
yield a black one (dotted lines) is l-^q. The probability of 
arriving at a given node in the graph, via a particular path, is 
the product of the probabilities associated with the. links in that^ 
path. The probability of arriving at a given node, irrespective 
of the path, is th^ sum of . the probabilities associated with all 
possible paths to that node. But every path leading t^o, a common ^ 
node has exactly the saihe probability of being, traverse^, because 
each is composed of the same combination of R and B links. 'So, . 
the easy way to calculate the prcfbability, of arriving at a node 
is to take the product of the probability of traversing any spe- 
cific path to that node and the number of paths leading to that node 
Figure 9 shows expressions for these probabilities for each 
of the nodes in our sample graph. In general, the probability eof 
arriving at a given node via a specific path composed ofm R l/nks 
. ^ and N-rm ,B 'links, is given by - 

q.d-q) 

and the probability of arriving at a node via any such path by ^ 

i 

The expected value of p(H^|D) after N observations, then, 
X. is giv4n by 

N 



/ > 



PN(HilD) 



where represents the posterior probability of H. after N 

observatiSns, m of which have yielded red chips. ^ 
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The following iterative formula can be used to compute /f 

(N-rm)q 



N,m+1 (m+1) (l-(5) ^N,m ' 



where 




and computation can be. simplified by taking logarithms: 

« 

, ' ^°5^N,m+l l*^'^ X -I- log Tf, 



N,m '\ / 



where 



\ 

(16) 



(17) 
/ 



(18) 



\ 



•'•°^*^N,0 " (l-q> 



(19) 



^^'■'4.u'^^u ^^iu^ °^ *3 equation (13) .de|)ends, pf course, on which 
or the :^ypotheses under consideration happens to be true. The 
expectation^ cain be computed, however, for each of the possibilities 
The general expression may be written as follows: / 



'[p«'Hll^fB,. is truej = fflP"'i«j)"P(BlHj)"-P,, (20,' 
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Figure 9-. Qraph illustrating^ the -computatipn of expected' 
, ■ values of p6sterior probabilities 

X- . ' (The expression above' each' node .represents 

.the probability of arriving" at that npde, given ' 
V / </ probability of drawing a red chip* The 

' • .^"f expected value* of the posterior probability 

following N observations is the sum of the values 
r ^ of'*the nqdeg above -N, each weighted by its 
"arrival" probability.)- • ' c 
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^ Figure ,10 shows E[pj^(H^|D)] for'our example (H^j, 70% red/ 
20% red),, given is true (top c»rve) and given' is not trxie 
(bottom curve) .* The top curve shov;s thfe expected growth of ' 
p(HijD) when is true an)a**of--p (H2ID) when is true. \ -Conversely, 
the bottom eurve represents the ex]^ecte4 decline of p{.H]^|D) when 
H2 is true and of pCHolD) when Hi is trug. Thus, in the^tifo- 
al^ter native case, . ^ 



. . * ' ^^'^^(Hj^lD) |fij^ / truej*'= E j^p(H2 |d) is' true j * (21) 

-To compute the. expected uncertainty following N ol servations, 
one must compute th^ uncertainty associated with each of ^the possible 
outcomes of the observations, and then take a weighted average of ^ 
. these .uncertainties, th^weights b^ing the pj^obabil^ities qf pccur- 

J rence of the speci.fic outcomes. The untrertainty associated with a 

• specific outcome, say .the outcopie N* observations yielding m red ^ 
.fchlps, 'is giV^en by ^ ^ ^ * ' 

. . ■ h - ^ * 

"N,m " " X^-^i;N,m ^^^2 ^i;N,in , ' . ' \ (22).' 

•J ' 

where h)is the number of hypotheses under consideration ^nd S^.^- 
is the probability associated with the "r*^^ hypothesis after ' ' 

observations yielding m red chips. The expected uncertainty 
after "N obsfervations, then*,' is obtained by weighting each U^j^j^ 
by its probability of occurrence, and summing over all 
possible outcomes. Thus,' / ^ 

W " LTTN^rnVm, ' (23) 

' % 

where' Tr„ > is*^ defined as before. Again, inasmuch efs the v41ue . 



of q in equation '0.3) depends on which hypothesis is true, the 
general expression for E(U ) conditional up6n which hypothesis 
is true may be written ' ^ , 



) 



E(Uj,|Hj is true) [l] p{R|H.)y(.B|Hj)N-^J^^^. (24) 
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• Figure, 10. Expected value of posjzerior probabi.lity H^, 
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•given Hj- iis true, as -a function of number 
of observations 
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V. The computation of expected ^uncertainty^ relates ^to figure 8 
ifi *the following way-. Imagine €hat such an outcome graph, were' ' 
developed for each of the hypotheses under considetatiorji,^ con- 
ditional upon the truth of a. specified hypothesis.,,* The vaiae of ' 

m found by sunilning .^-p iog2p across gra*ph;s^, f or given N^m, 
tn& values of 'p being the values of ^the nqdes on the g^phs * 
(equation 22) • The value of E(U ) is thep obtained by weighting 
each of ±hese ^ums by ^the probability of arriving at node N,m, 
given the £ruth ^of the specified' hypothesis, and summing over m, 
(equation 23) . Figure 11 shows how the expected uncertainty con- 
cerning which of the two hypotheses is true changes as a result of 
observations in the case of our example* (H.: 70% red, H^: 20% red)*, 
given the- .truth of each hypothesis in turn". ^ 

The examples that, we have been considering n^av^e all converged ^ 
rather quickij^ to a stat^ of- relatively low unce|t:tainty . This was 
due to the fact that H, ^pd were quite dispatate. But suppose / 

and H2 were sindlar witji r^pect to their .implications for 
data. Suppose, for^xample, that we let H, be the sair^ as before 
--(that is, that 70% of the chips are red) and H'^ be the\hypothesis 
that 60% of the, chips are'r^d. Again, setting the initial .probd- , 
Ijj^Iities ecjual to and assuming the same sequence of observations 
as iridicatecf in figure 1, figures 12 and 13 show the effects of 
these bbservations on the distribution of probabilitieg over the 
two hypotheses, and on uncertainty. Figures 14 -^nd 15 show ^the 
ex pec ted effects of* dat^ on postjerior probabilities and uncertainty 
for this case,.' Obviously, the" eiipected,*' effects ^6f observations 
are mucji smaller — the data have less diagnostic impactj*^-wjjen the , 
hypotheses are similar th^n when they are very different • Or,, to 
say the fe^e thing in other words, a larger sample is needed to 
.produce thVsame degree of certainty^ith respect to which .hypo-, 
th^esis. is" tnue* This illustrates t^e intuitively qompelling idea 
•that the smaller the differences between two statistical distribu- 
tions, the closer one must examine?^ them to tell which is which.. 
Continued sampling vdll ejj/entually make the probabij^^ities diverge 
and the uncertainty decrease, assuming, of course, that the samp- 
ling is random ^nd one of the hypotheses, is in fact true. Figures 
16 .an'a* 17 show th^e^e^pected changes in p(H^[D) and uncertainty 
over the course ot 200 observations", giyen H, : 7Q'% red, H^: 60% ifed^ 
Pq(H*) = .^'^^ Two hundred observations would not, on the average, 
reduce the uncertainty in this case Ijhe. amount that ten ob- 
servations would reduce it, ^given the more disparate hypotheses, 
: 70% red "and H^: 20% fed.' 

Table 2 (page 95) Shows, for various combinations of and H^/ 
the expected posterior probabijfitY of H. after ten observations, gi 
that chips are, sampled *f rom an uigri containing reds and l^l^cks in . 
the proportions specified by H^, 'and ^assuming *Pq1h^) = Po(^2^ 
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Figure li." Expected uncertaihty concerning which 
hypothesis is tru,e- as a function of 
' number of observ^-hr.ons " ' 
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Figure 12.; Changes in. posterior probabilities, 'p (H^ | D) 
^ and pi^li^lD) r. as a result of the indicated 
-observations of Red and Black chips 
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• Figure 19^. Expected uncertainty concerning which 
hypothesis is true ,oa^^ function of 
mjmber of observatic(as 
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are several "things to notice about this, table. First " 

.from-W-urn for which eA^^^"^ ^ ° Jhe'^a'JplfT?''" • 
taken- from an ur.n for" wEich H-, is' truef A second Jhi-m^ Ei t- ' 

llTatnT '^'^ '^'^ is syl:;r?S'abo'u"?^h'e S^no^ 

diagonal Thus for example, the probability associated with ?L 
true hypothesis is the same for H^: x% red, L: y% Jelas fo^ ' 
Hi: y% red, H2: red. , i^his is I trivial poLtraSd simpl? ' 

. indicates, ^hat the- expected effect of a Sequence of obser^^?inn = 
IS striagy a function of the-diagnosticiSJ o? Ia?f anl S inde- " 
pendent ql which hypothesis is which/ Third, except wSen onfo? the 
hypotheses IS extreme (say, hypothesizes that 10% or iSss or 90% 
data Il'laf^^v ^^JP^' one color), the expecjted imp^c? of 
d^ta IS largely a fuaction of the difference between the hvooShP^ 

.^agStSirs?"'"""' relatively independent of " thJLt^oE?^ ' 

.ni H^^' Zl^ ^^"""^ ^^"^^ various combinations of H. 

.the first one or two observations can have a remarkably 
-lar^ effect. Hov^ much effect, they will have depends, howe^^er - 
on wh*.t those observations are and <^n-the disparity b4tweenT' ^ 
and H.2. This point is illustrated by figure 18, The figure shows 
-2d Hi,:; given a sihgle' observation tSat yields a 

red chip. In all cases.,^ it is assumed that the hypotheses were 
equally probable before the observation. Note thatif the hypotheses 
are clisparate for example. Hi: 90% red and H,: 10% red? or 10% 
red and . H2: 90% red, a .single^obfeerva'tion wilf change the profc- 
bilities associated with Hi and^ from .5 and .5 t6 .9 and 1 
or to .1 and .9. On thfe other hand, if the initial probabilities 
?SL"ve?y1??;ie"^".-' '""^'''.^ single., observation Sill change"' . 



8.6.8 The Symmetrical. Two-Hypothesis Case 



nno J two-hypothesis case of special interest is that- for which 
one of -two possible observations has- the same probability qivS 
one hypothe'sis .as does the other observation given the oJheJ 

pirif) •-•o\d'|^ I"' concerned with tL .ituatiS^IS which 

P(D^^.|H.^) - P(Dg|H2,).,-or equivalently, .m^whlch P(D^.|H^) = l-p.(.D^ | . 

referred tQ as the " symmetrical case, reflecting 
the fact that one of th^e two possible observatiorfs'provides eSac^l? 
as much supi^ort fbr one of i^e hypotheses as does the o'^hSr obS^- « 
vation for the other hypothesis.' This situation holls i-n ^he 
chips-m-urn context when both hypotheses involve the same d?o- 

?:fch?ps tnd'tSf oli^' \if^--t colors, Sut on J id^nt^fLs 
r^d chips, and the other black-chips, as. being the moire numerous. 
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Figuire 18* The effect of drawing a single Red chip, 
given various combinations of prior^ 
hypotheses concerning the proportion 
j3f Reds in the urn 
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The hypothesis pair H^: 70% R, 3.0% -B; H^*: 30% R, 70% -B satisfies 
thi'S condition, 'for example; whereas th4 pair H,,: 70% R, 30% B; 
.H,: 20% R, '80% B does not. ' ^\ ■ ' 

The si^etrical case is of special interest because of tJie • 
fact that, the effect of a series of observations .on the odds-'tavor- 
i'ng on^ hypothesis over i:hfe other can be calculated in a trivially 
^simple way. If represents the odds prior to the observations 
of interest., and L represents the likelihood ratio, then 

* where d represents the difference between the number of observa- 
tions of -{say, red chips) and^brf'D (say, black chips) , and - 
represents the odds following the observations. Note that the 
size of the. sample — the number of observations — does nou enter into 
tHi^, ^calculation* Suppose,* for example, that = l,and L = 3 
(as would be the case if p{R|H^J = .75 and p(R|h ) = .25, and the 
odds and likelihood ratio were expressed relative to H>,) , then, 
given a sequencejof observ^'^tions'yie.lding lour more red cfiips 
than black chipST thq^ostetior, odds would be , , ' 

*• • * . 

fi, = 3^*1 =81,' (26) 

ancl' the same result would hold whether the difference of four was 
obtained from a j^ample containing 8 reds and 4 blacks ,or one con- 
taining 100 re^s and 9.6 hjlacks. 

The exclusive dependence of , on d follows directly from " 
the fact that the likelihood ratio for one of the two possible ob- 
servations is the reciprocal of that for the other observation. 
Recall from e<}Uiation ^(11)* that the posterior odds following a 
single 6bservatio;i is simply tne prior odds multiplied by -the 
likelihood ratio associated with th'e observation 

Recall, too, however, that the lilcelihood ratio is conditional on 
the observation. Thus, if D is observed, 

.p(D^ 

> • . ■ . 

« 

"Whereas, if Er^ is observed, / 

L - • > (28) 
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Letting represent the likelihood ratio when is observed, 
and L^^the likelihood ratio when Dg is observed, we may represent 
the effect <ac a Specific sequence of observations, say 



D D D^D'D D an thfe odds as 
- a a £ a 3 



In the syiranebrical case, however, 

i \ ■ 

so the effect of the same specific sequence of observations may 
De written as * • ^ 



and in general 



n a n-k * ^ * 

* 

X 

where d is the number of observations of Da minus* the nunfber of 

.observations of D3. But, inasmuch as neither n nor k is used in 

the calculation, we may express as a function of d, and write 
the expression as in equation (55). • , / " wxj.i.e 

We see then that in the symmetrical case, the odds increase 
exponentially with ^hg difference between the number of observa- 
tio^is -of the one ty^>e and that of the other typfe that have been r«. 
obtained. Figure 19 ^hows how the rate of growth of this function 
depends on the disparity between the conditionstl probabilities or 
equivalently, on the size of the likelihood ratio. Figure 20 ' 
shows how the size of the difference that is i^equired to realiz4 
^^''u y^^^®? ^i^h the lairge^r of the coridi^y.onal probabilities 
of which the likelihood ratio is' comprised, frhe finding that peoplS^" 
typically tend to be conservative Bayesians in their use of data 
to revise their • estimates of the likelihoods of the {possible states ' 
of the world suggests that many people would find the relationships ' 
that are shown m the'se figures to be counterintuitive. The fact , 
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Figure 19. The ra^fe^of growth in posterior odds as a 
function of- the difference/ d, between 
the' number of .observations favoring 
• and the numlz^r favorirtg in the 
symmetrica/ cas>2 ^ (The parameter is 
likelihooc^ ratio, 2^ 
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Figure 20, The size of d reqCiired to realize a given 
odds ratio as a function, of the larger 'of 
the two conditional probabilities in the 
symmetrical case 



ERIC . 



101 



•.114' 



NAVTRAEQUIPCfiN 7 3 -C- 01 28-1 - 



0 



« f or exampl,e, that with conditional probabilities of p(p jH^^) = .7 ^^ 
and*'p(D iRj) = !.3, a sample that contairl^^s three more observations of 
Da than^^of ' Do will favor 'H, over ^2 by a factor of rrtore than 10 , 
^lay.'be surprising; as may €4ie facf^that wi'th a difference of six, 
the odds are, greater, than 100 to 1. < ^ . 

Another , aspect of the' syitunetrical case that some readers may 
find counterintuitive is the fact that the total effect Of a series 
of observations on the .odds depends only on the difference between 
the number of 'observations of the two types and is independent of 
* the total number of observations' made. Both intui^tion and statis- 
tical training suggest that one's confidence in any inference that 
is to be drawn from the properties of a sample should ^J.ncrease with 
the^sample. size. The apparent paradok is resolvied hy^ a recogni-ti^n 
of the fact that/ except under the hypothesis that^ ea^h observation 
is equally likely, the absolute difference (thbligh not' uhe relative 
difference) between the frequertcies of occurrence of th^ two types 
of observation is expected to increase 'with sample size. Speci-' • 
fically, if H: x% (l-^x)% B is true, the difference between the 
number of Rs -and Bs in a sample of siz,e N should be (2x-l)N, on ' 
the average. ^ 

Consider, 'for example, the symmetrical hypotheses H : 70% R, 
3 0% B and : 30% ' R, 70%' B. If were true, samples of ten dpws 
would be expected to produce four more ♦reds th^an blacks on the' , 
average; and. the odds following a ten-draw sample with four more^ 
reds than blacks ^would be 'about 30 to 1 in favor of H^^. ^ Samples of 
100 draws, given H-,. should produce 40 more reds than blacks, on 
the average., 'a. difference that would, drive the odds to more than/ 
523 trillion to 1. Thus, the odds dp tend to increase with sample 
size because, d tends to increase with sample size. A sample of 
100 draws that prdduced four more reds than bladks would be quite 
.unlikely if H-. were tJjue, and thus would not constitute strong 
evidence in favor of tha^^t hypothesis. It would be even less 
likely,, however, if H2 were true, so it c^oes constitute some 
evidence for Hq^,^ but only as muC^h as one would ex£iect to obtain 
from a much smaller sample. TaMe 3 shows the odds favoring H^, 
given various combinations of p(p|H. ) and p(d|h^) and several 
values of d. ^ \ , ' ^ 

8.6. 9 The Several-Hypothesis Case ^ ^ 

So far, the examples that we have considered tdxillustrate 
the tise of Hayes rule have involved only two hypotheses. We turn 
now to cQnsideration of a few cases in which the^e ar4 more than 
two hypotheses. Figure 21 illustrates a, case in whictr-H-j^, H2 
and Ho represent, respectively, the hypotheses that the percentage 
of red chips in the urn is 90, 70 and 50, and shows how the pos- 
' terior probabilities associated witth these hypotheses would change 



O 102 



NAVTRAEQUIPCEN 73-C-0128-1 



TABLE. 3. ODDS FAypRING GIVEN THE INDICATED VALUES OF 
^ . ^. • p(D|H^), p(DiH2) AND d. v 
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All odds varvies are rounded to two significant digits and expressed 
in exponential 'form. To obtain the approximate value of fl, multiply 
the nvimber to the "left of the E by ten raised to the power indicated 
by the number to the right of the E. For example, 4.3E7 = 4.3 x 10*7) 
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R R B .B R '. R B ' R R R 

. . DATUM 



100 laa 67 50 60 67 59 62 67 70 
. PERCENT OF REDSKIN SAMPLE 

H^: 90% Red; U^: 70% RecT; 
C H3: 50% Red;^pQ(H^) = .333 , t 

'/ 

Figure 21. Changes in posterior probabilities p(H.|D) 
as ^:^esult of ' the indicated, observations 
of Red and Black chips 
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as a result of th^ indicated sequence of observations, given p^(H.) 

= .333. Figure 22 shows the change in uncertainty concerning ° ^ 

which hypothesis is ^correct as a result of the s^me sequence of 

oBservations . ' , • ♦ 

7 

When only tv;o hypotheses ^aj'e u'rlder consideration, there is . / 
only^one odds ratio (or its reciprocal) that can be expressed. ; 
The nflmber of odds ratios that can be expressed grows rapidly, 
howevei:, as the number, of hypotheses is increased beyond two. In 

general, given N hyfJotheses, tbere are[2| or N(N-l)/2 odds, that 

can be expressed considering only pairs of hypotheses. Thus in t^e 
three-hypothesis case we might consider fi, ^, or ^, each • 

of which is shown in- figure 23 for our example. 

It may be of interest to consider bther than pairwise odds ^ 
ratios^ in the several-alt&rnati^e case,^ however. Given five ^ 
hyJ)Otheses, .for example, one might wish ifp consider the odds pf 
H]^ relative to the combination of H3 and H4, which would be oiD- 
tained by taking^the ratio of p(Hj^|D) to the sum of ptH3|D) and 
pCH^fo). It may often be of particular' Interest to consider the 
odds -of a given hypothesis, relative to all the remaining 
hypotheses in combination. Such an odds would give the ratio of 
the probability that \Hi is true to the .pjrobabilijby that one of 
the remaining hypotheses is true, i.e., that Ua is false. We ^ 
might refer to such aji oSds as the absolute pddc of Hj^ and repre- 
sent it as follows: < ^ ^ 



P(H.JD) ^ p(H^|d) 
IT]D) F^pTSt] 



"i,i -PCH.ID) l-p{H. |D) ^23^ 



^ Figure 24 shows how the indicated ohsefvations affect the ab- 
solute odds of each of the hypotheses of ''our example. 

Expected values of posterioi probabilities and of uncertainty ' 
may be calculated in the same way when there are seve^;al hypotheses" 
as when there are only two. ] An outcome graph' such as those shown 
r figured 8 and 9 could be, used to specify all possible posterior 

\^ probabilities for a given hypqthesis, H . , and their probabilities 
of attainment on the assumption that a- specified hypothesis, H. is 
true. The weighted sum of the nodes above a particular yalue ^ 
of N-^would represent, as before, tMe expected value, after N obser- 
. yations, of the posterior probability of , .given that is really 
true. Also as before^ computation of expected uncertainty involves 
summing over both i and, m, for given N. Inasmuch as it is possible 
to compute an expectation of p(H^|D) given that H^" is true fbr all 
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Figure 22, Changes in uncertainty as a result of 
, indicated obseryatibns 
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PERCEfviT OF REDS 7N SAMPLE 

- ■ 

Hj^: ^0% Red; U^i 70%. Red; q ^ 

' "3*„^°* Po^^i^ =.333 

Figure 23. Changes in 'pairwise odds as a result of 
the indicated observtations 
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Figure 24i^- Changes in absolutte oddsvfor -each hypothesis 
as .a tesult of' the' indicated ^observations 
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possibl'e combinations 6f 'values of i and j, the number jo§ putsoipe - 
graphs that coul4/be of interest increases with the square of the 
number of hypoti^ses under '^consideration. _ , \ 

. ~ Table 4 shows the" fekpected values of p^di. |d) and U , 'fk>x s 
N = 1,2, ...ro, given that Hj is true for afl Qombinations^of 1 
and j in the case of our example (H, : 90% rfed,., H^V 70% red. Hi*. 50% 

p, '" ^ - * 1 . . - . 7 . 'Si. 



S 



red, Pq(H.) = .333).^ As might be expected, given that H.: isnflcrud, 
E[p(H^TD)r increases most rabidly when i = j;. which is t6 iay , . 
th6 expected value of the' probability associated with the true 
hypothesis grows faster than that of the probability associated' 
with either false hypothesis. Counter to intuition, ho^eve^r, this 
is not a necessary condition. An example will be . considered 
ptesently in which the expected probability as^sociated. with a fal^e" 
hypothesis grows^ for a while, at a greater rite, than does 'the 
_ expected iprobability associated with the true hypothesis,, even 
when both hypotheses are equally probable a priori. , ith continued 
sampling, however, the probability of the true hypothesis evfentually 
gets larger than that of any of the false hypotheses Another point 
of interest 'concerning table 4 is the fact that eacif of three 
columns of values occurs twice: the second and foucth columns 
are -aJS^aptical, as are the third and seventh, and the sixth an^ eighth. 
This iriustrateys the ^dll^ving relationship: ■ 

^ Etp(H^|D)|H. is true] = Etp(H. |D)1h. is true] , '(34)' 

that is, the expected posterior probability of H^, given that H. 
is true, is the same as the expected posterior probability of . ^ 
,Hj, giyen that is true. This .relationship holds in general, and 
iiidependend^ly of the number of hypotheses under consideration. 

As in the two-alternative case', th^ rate -at which the expected 
values of the posterior probabilities approach pne or zero — and, 
consequently, the rate at which uncertainty is expeqted to decrease — 
depends on the dispatity among the hypotheses. The poTnt is illus- 
trated in table 5, which shpvs all values of E [p, q (H • | D) | ' is. ' 
true] for two sets of hypotheses: Hi, H2 and H3':,?0,\o and 5^^% 
red, and'^O,' 60 and 30% red. The table' also shows the expected- 
uncertainty after ten observations, E(Uio')/ concerning which 
hypothesis is trqe, as a function of which hypothesis actually is 
true. . . ■ - 

Table. 6 shows E [pio (Hi |d) | Hj is true] and ECUiqIh-? is true)" 
for two sets of five hypotheses. This table illustrates some of the 
same points as does table 4. The rate at which the probabilities 
change from their original values, and the rate at which uncertainty' 
decreases depend on thg disparity among the hypotheses. The value 
of Etp(Hi|^J.[H-j is' true] is alway-s equal to that of E.tp (Hj | D) | Hi 
IS- true] , whicn is seen by the fact that each array, if considered 
as a matrix, is equal* "to its' transpose. 
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^ TABLE 4. 



EXPECTED VALU:^S OP POSTERIOR PRQBABILITIES , AND 
UNCERTAINTY (IN BLTS) , GIVEN THAT CHIPS ARE SAI-IPLED 

FROM THE URN FOR mUCB THE' INDICATED HYPOTHESIS 

■* ■ ■ » 

IS TRUE. 



frue Hypothesis* 



Hypothesis^for which Expectation Computed 



I 

to 




«1 


«2 • 


"3 


Er(U) 


«i : 


«2 


«3 




H 


«2 


«3 


E(U) 


1 


.397 


.333 


^270 


1.53 


.333 


.333 


.333*^ 


1.49 


• 270 


.333 


..397 


1.4^ 


C 
0 


2 


.453 


.327 


.220 


a. 44 


^.327 


.338 


.33.4 


1.41 


.220 


• 334 


.446 


1.35 


•H 


3 


.501 


.319 

r 


.18a 


1.35^ 


.319 


.347 


.334 


1.36 


.180 


.334 


.486 


1.26» - 


> 
U 


4 


. 542 


.310 


.149 


1.26 


.310 


.358 


.332 


1.32 


.149 


• 

.332 


.51'9 


1.19 


o 
to 


5 
• 


.577 


.300 


.123 


1,18 


.300 


.370 


.330 


1.28 


.123 


^.330 


.547 


1.12 


o 


6 


.607 


.291 


.102 


1.10 


.291 . 


.383 


• 326 


1.25 


• 102 


.326.->^72 


1.06 




7 


^634 


.281 


.084 


1.03 


.isi 


.396 


..322 


1.22 


.084 


.322 


.593 


1.00 




8 


.658- 


.272 


.070 


. 0.96 


.272 


.411 


.318 


1.19 


.070 


• 31.8 


^ .612 


0.95 


1. 


9 


.680 


'262f 


.058 


0.9a 


• 262 


.425 


.312 


1.17 


.058 


• 312 


' .630 


d;9i 




10 


.699 


.253 


1 04 8 


0.85 


.253 


.440 


• 307 


1.14 


.048 


.307 


.645 


0.87 



Hj^: ^% red, 70% red, H^: '50% red; Pq(H^) = .333. 

(Noce: ^ expected uncertainty, E(U), ds not the same as the 
uncertainty calculated from the expected posterior probabilities. 
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TABLE 5. 'E[p^q(H. 1D) IhJ IS TRUE] FOR COMBINATIONS OF 

■ • ■ i, and j^ND THE TWO INDICATED HYPOTHESIS SETS. 

H^i 90%, Red, H^z 70%- Red/ H3; 50% Re^ ^ 



* 












^ 




* 

/ 1...' 






3 




* 

1 


- .699/ 




.253 ■• 


.048 


r 

i 


.2 


.253 




.440 


.307 




3 


.048 




.3D7 


.645 


E{U) 




0.85 




1.14 


0.87 


(in bits) 
















90% 


Red, 


H2: 60% Red, 


H3: 30% 'Red 










j 








1 




2 . 


' 3 






.824 




.171 


.005 


i 


■ 2 


.171 




.603 


.226 




3 


.005 




.226 


.769 


< 

E(U) 




0.48 




0.86 


0.55 ' 


(in bits) 












In 


both 


cases Pq 




= .333. 


• 
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TABLE 6. E[p^Q (H^|D) |Hj IS TRUE] FOR TWO FIVE-HYPOTHESIS SETS 







90, 80, 








• 


H1-H5: 


70, 60, 50% 


Red, respectively 




1 


2 


3 


4 


5 


1 


.475 


) .282 


.148 


.068 


: .026 




.282 , 


.^71 


. 216 


.146 


.085 


i 3. 


.148 


.216 

/ 


.239 • 


.221 


. .176 


4 


.068 


.14ff 


.221' 


.273 


.292 


• /5 


.026 


.085 


.176 


.292 


J .421 


E(U) 


1.64 


1.90 , 


1.96 


1. 86 


1.67 


(in bits) 


■ 












H^-H^: 


90, 75, 


60v 45, 30% 


Red, respectiveli 




t 


. . ' 


< j 








1 . 


2 






5 ■ 


1 


.604 


.280 


.094 


.021 


.002 


2 


.280 


■ .337, 


.240 


.112 


.031 


^ 3 


.094 


.240 


.3-00 


.240 


.126 


4 


.021 


.11-2 


.246 


.323 


.304 


5 


.002 


.031 


.126 


.304 


■ .537 


E(U) 


1.19 


1.63' 


1.75, 


1. 63 


1.30 ' 



(in bits) 



In both cases p-.(H.) = ^2. 
• u 1 
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For the hypothesis set .represented by the bottom half of 
table 6, it is true that the expectation is maximum when i = j, 
which is to say that after ten observations the probability of 
the true hypothesis is always larger than that of any of the false 
ones. Note, howeyer, that tKis property does not characterize the 
values for the* hypothesis set represented by the top half of the 
table.' In particular, given this hypothesis set, the expected 
probability of Hj^ is greater than that of H2 after ten pbservatibns, 
-even if chips are drawn from an ^urn for which is "^rue. Similarly, 
E)(p, q(H- |D) ] is gi^eater 'than E[p,q\H. |D) ] when H4 is true. With 
conrinued sampling the expected probability of the true hypothesis 
v;ill continue to grow, finally approaching one, whereas that of, 
each of the false hypotheses will at some point begin to decrease 
, and will eventually approach zero. The fact that the expected , 
value of the probability of a false hypothesis is higher at any 
time than that of the true hypothesis may' be quite counterintuitive, 
however. Figure 25 shows the way in which the expected values/ 
of each of the posterior probabilities of the example represented 
in the top" half of table 6 change over twenty observation^, given 
that .is-^really true. .Note that p(H2|D) is initially smaller 
than pTH^|D), but eventually overtakes and surpasses it; with., 
further Sampling p(H^|D) would continue to increase, whereas 
p(Ht |I>) would decrease. 

A comparison of tables 5 and 6 xillustrates several additional 
points. The hypothesis sets represented in table 5 are contained 
within those represented in table 6., Considering only those, 
hypotheses that are represented in both tables, it may be seen 
that the expected posterior probabilities associated with hypotheses 
within the smaller set are invariably larger than those associated 
with the same hypotheses within the larger set.. It may also/be 
seen that the expected amount of uncertainty remaining after ten 
observations, given the truth of *,a specific hypothesijs, is ^greater 
when the hypothesis set contains f ^.ve^ alternatives than when it 
contains tl>ree. Of course, the a priori uncertainty is also 
greater in the former case (2.32 bits versus 1.58 bits), so,' what 
is of greater significance is the fact that 'a larger proportion of 
the original uncertainty is resolved in the three-alternatiye case. 
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H^: 90% Red; U^i 80% Red; 
H^: 70% Red; H^: 60% Red; 
Hgi 50% Red; Pq (H^). = ^2 



V 

Fi'gure 25. 



Expected value* of posterior probability of 
H^, given that is true", as a function of 
number of observations 
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8.6.10 Man as a Bayesian Hypothesis Evaluator 



< 

A considerable amount of ekperimentation has been done to 

duaCSLnS ""'^ P^^'^'^^^ ^"^h^-^^- when an inliv?- ' 

Tiil ?Ssf niust?a?2'' probabilistic information in situations 
iiJ-^- Illustrated. For example, given the task of 

deciding on the basis of a sequence of observations/ which of 
several hypotheses about the nature of the source of those ob- 
servations IS true, how closely will the\ estimates produced by the 
human decisionmaker correspond to those produced by the application 
of Bayes theorem? Obviously, in situations as highly structured as 
those described, it would be of litt,le interest to do such experi- 
ments with an individual who linderstood Bayes rule and was permitted 
to do the calculations necessary to use it. Such a test would do 
nothing but demonstrate one's ability to do -arithmet-i-' Experi- 
ments on Bayesian information processing typically ak'done with 
people who. are not formally aware of Bay^s rule, or if they -are, 
•they are not provided with the time to perform the necessary calcu- 
lations. It IS an iRteres;ting question, in this case, whether an 
in rtf^S J ^S^^^J^^^r at least informal, notions about evidence 

Perhaps the question that has been of greatest interest i-n 
and received most attention from, experimenters is wJethe? hvpoiheses 
are more effectively evaluated by having decision mrklrs LtimaS 
posterior probabilities,, p(h|d) , directly upo? acquSng !ncom?na 
iTtyZ In estimate conditional probabilitiel, p^STh?? 

S??h ^ho ° ""^t n^^^ estimates to update the posterior probabilities 
with the use of Bayes rule. Much of the evidence favors JhP^nn 
elusion that hypotheses are evaluated more eil?cien?I? when t£e 

btslH) ^nfSSL'' J-^'J' "^^^"^ ^^"^ ^^"^^^^ "'ake estimates Of 
P(Ei|H) and these estimates are used along with Bayes rule to cal- 
culate estimates of p(h|d) . Although thi directional ef?ec?s of 
si^tltr lo^ltnii'' probability estimates produced by humans a?e - 
similar to those on estimates revised in accordance with Baves rule 

ln%lrlTcllst' ?h'.''' f ^^^^^^^ Se fSer casl.' 
in particular, the posterior probabilities tend to obtain more ev- 

t^Z%y^^''t^ t^*^ ^° ^^^^^ asymptote faster when they aS callufated' 
accordmg to Bayes theorem than when they are estimated direc?lv 
by humans (Edwards, Lindman, & Phillips, 1965; Howell f Ge??5 1968- 
Kapi^n & Newman, 1963; Peterson & DuCharme, 1967rieterson ?MiLor 
1965; Peterson, Schneider, & Miller 1965- Phi nn-r.= r^^ ^ ,0^5' 
It appears, therefore, that humans tend' 'to exiJacriesf S?a?matfon' 

iT ''^•^ ""^^ ^^""^ contain; they require Se ev^lencf ?San 
does a: Bayesian process to arrive at a given level prclrJIinJ? • 
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concex^ning which of the competing hypotheses is true. That is one 
of the findings that has led to the characterization of man as a 

conservative" Bayesian. In other words, men tend to underestimate 
high posterior probabilities and overestimate low ones. A similar, 
but -less pronounced, tendency is found when men estimate odds 
rather than posterior probabilities (Phillips & Edwards, 1966). * 

Slovic and Lichtenstein (1971)- refer to* the conservatism of 
man in his use of probabilistic data as the primary finding of 
Bayesian research. They review three competing explanations of 
the result: (1) misperception, or misunderstanding, by the subject 
of the process by which the data are generated; (2) inability of 
subjects to aggregate, or put together, the impacts of several 
agtato produce a single response; and (3) an inability, or un- 
willingness, to assigrr extreme odds, e.g., odds outside the range , 
of 1:10 to 10:1. Whether any of 'these explanations is adequate 
has yet to be determined. ^ 

^^u-"^® finding of conservatis^n that"prQmpted Edwards (1963, 
- - 1965) and his colleagues (Edwards, Phillips, Hays, &' Goodman, 1968) 
to experiment with probabilistic inf ormation-prbcessing systems 
.that use experts to judge the- likelihoods of the data reachinq the 
system, given each hypothesis under -consideration, and machines 
to calculate posterior probabilities on the basis of these estimates 
.ana tne aata# 

Not all of the evidence -that is relevant to the question favors 
7Mm''S''^''°'} K^f^ humans are invariably much better" at estimating 
P(D|H) than pjHfD). Southard, Schum, and Briggs (i964b) , for 
. example, obtained some results that .challenge £he* generality of the 
finding that humans tend tb underestimate high posterior proba- 
bilities, and overestimate low ones. In particular, given a small 
^S?^?^^''^^^^.^'"^ ^ frequentistic environment, the estimates of 
?u ' -^^.^^ humans wire lose to, and sometimes more extreme 

than, those produced by Bayesian methods, other studies, several 
from the same laboratory, have also yielded results that question 
the validity of the general conclusion that better decisions result 
ooSLIJ ,g S ° ^""^ tierived by applying Bayes rule to men's 

estimates -of p(D|H) (schum, Goldstein, & Southard, 1966; Hovrell, 1967; 
Kaplan & Newman, 1966; Southard, Schum, & Briggs, 1964a) . Often 
even when evidence of conservafcism has been -found, "the degree to 
■ which the human's estimate .of p(h|d) has differed from an estimate 
produced by Bayes rule has been very slight (Peters0n & Phillips, 
1966; Schum, Southard, & Wombolt, 1969). / 

These findings do not' permit one to conclude that' estimates 
Of p,(H|D) are never better wjien derived from estimates of p{DlH) 
than when produced directly, but they do call into question the 
. opposite notion, namely' that of the invariable superiority of the 
indirect approach. Moreover, they suggest that the direction that 
reseaijch should take i^ that of determining the conditions under 
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which each approach is warranted. ^ Schum, Goldstein, and 
Southard (1966) ' oresent some data, for 'example, that suggest' that 
estimates ofp(H|D) that are produced directly are more adversely 
affected by degradation in ^ the fidelity of -the incoming information 
than are those tha^: are derived from estimates of p(D|h) . 

Another finding that is relevant to the^question of man's 
capabilities as a Bayesian hypothesis evalua^:or is that evidence 
that tends to confirm a favored hypothesis may be given more 
credence than evidence that tends to disconfirm.it (Brfedy, 1965; 
Geiler & Pitz, 1968; -Pitz, Downing, & Reinhold, 19'67; Slovic ^ 
1966). This finding raises the more general question of whether 
a vested interest in a decision outcome impairs *one'*s ability to 
evaluate data objectively. » If it'is the case, as Bacon (1955) v 
long ago suggested, that "what a man. had rather were true, that 
he more readily believes," at least- one of Savatcfe's basic 
rules 'for the application of decision theory is gene: ally violalied. 

The possibility that an individual ' s .preferences among 
hypotheses may impair his ability to evaluate them in an unb-iased 
way is closely related to the finding that fleople tend to be 
reluctant to change a 4ecision once it has been made (see Section 
-•4.3) . ■ . • : ' . 

'Each of these tendencies — conservatism, partiality, and 
pqrseverativjeness — has been viewed as a fault, or as, evidence 
that 'man applies data to the^^aluation of hypotheses in. an 
inefficient way. And, in t*[^context of most laboratory experi- 
ments in which it has been observed, it' undoubtedly is. These 
tendencies may sometimes be less patently un.justif iable outside 
the laboratory, however. An insistence on having cQp^pelling 
evidence before changing an established opinion may have a 
stabilizing effect that is not altogether bad. Many opinions 
re fomed slowly ov^r a period of years, and all the factors 
hat may have contributed to their formation cannot Always be 
recalled at will. The individual who is quick to change an 
opinion every time*he encounters an argument that he cannot 
immediately refute^may find himself constantly shifting from one 
position to another, always a proponent of the view that he last 
heard capably expounded. . ^ 

Hypothesis evaluation has been studijed more than most aspects 
of decision making in the laboratory. This is due in part to the 
existence of a simple prescriptive model (Bayes rule) for per- 
forming this task, given an appropriately structured problem, and 
in part to the fact that it lends itself to laboratory exploration 
more readily than some of the other decision -making functions/ 
Much has been learned about mah's capabilities and limitations 
in applying evidence to the resolution of uncertainties about the 
various aspects of a decision situation. Much remains to be • 
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determined, however, sfnong several issues mat deserve further 
study are the following: the possibility^Mat information-display 
formats' and response techniques may ifl^iffen^e subjec.tive proba^ 
bility estimates that aire obtained {I>^j^<^Goo^man, ^Peterson,, 
19^72; Herman, Ornstein, & Bahrick, 1964); the appardnt lack of 
understanding of how to combine probabilities arising 'from inde- 
pendent sources of information (Fleming, 1970); the possibility 
that the weight that one attaches to data may depend on when those 
data occur during the hypothesis-evaluation process (Chenzoff, 
Crittenden, Flores, Frances, Mackworth, &. Tolcott, 1960; Dale, 
1968; Peterson & DuCharme, 1967); and. the possibility that one's " 
ability to deal with uncertainty in a conflict situation may de- 
pend on^ whether one is operating with an advantage or a disadvan- 
tage with respect to one ' s. opponent (Sidorsky & Simoneau, 197 0). / 

8.6.11' Bayesian Hypothesis Ev^lua^ion and Training 

One way to interpret some of the results that liave been de- 
scribed above — for example, the finding, that men often extiract - ' 
less information from data ^an does a Bayesian aggregator — is 
to see them as indications that man's intuitive notions concerning 
the uses of evidence are not entirel^^ consistent with the implica- 
tions of Bayes rule. Perhaps the thing to do, if this is^ the case, 
is to disabuse would-be decisiori makers of those faulty intuitipns^^ 

Such a task might be appifoached in t\^o v;ay's. On the one 
hand is the cognitive approach of teaching the decision maker about 
Bayes riile and its implications. An alternative possibility is 
to expose the decision maker to a variety of situations, in which 
his behavior is evaluated and immediate feedback is provided to'" 
him concerning the way in which it departs from optimality,. if it 
does. This is the behavior-shaping approach; in essence, it is 
aimed at modifying one' s - intuitions without necessarily providing ' 
an intellectual understanding ^f how optimality is defined. These 
two approaches are not mutually exclusive, of course, and it seems 
reasonable to assurtie that a training program would ,be more likely • 
to be effective if it used both. That is to say, the decision 
maker should ^probably be given a good understanding of tjhe notion 
of inverse probability and how Bayes rule aggregates data; and he 
should also be provided with considerable practice in attempting 
to apply the rule in situations -^hat are sufficiently well-struc^ ' 
tured that his performance ^an/be evaluated and "compared to an 
objective criterion of optimal ty. The selecti6n of training 
scenarios should put special emphasis on those situations for 
which man's intuitions have bden shown to be most misleading, e.g., 
especially small or especially large levels of a priori uncertainty 
and situations in whici) the direction of evidence changes after 
a tentative decision has been ^reached. 



ERIC 



118 

* 

131 



NAVTRAEQUIPCEN 73-C?^Qa28-l 




The results of some studies have indicated that such -training^ 
can be at least partially effective. Fleming (19,70), for example/ 
explored the que^ion of the effectiveness of feedback concerning 
the outcome of a selected action in improving th^ decision maker's 
performance on subsequent decision tasks. The context of the 
study was "a simulated tactical decision-making situation. Subjects 
were required to combine probabilistic data from three independent 
sources in order to arrive at an estimate of the relative' likeli- 
hood of attack on each of three ships. Initially;^ subjects demon- 
strated an" ignorance, o^f the proper combining rule (multiplication) 
'and were conservative in estimating the overall probabilities of 
attack. The investigator concluded that these data-aggregation 
and^probabilityr^estimation tasks shoul'd be automated. He also 
shoWed, however, that, although the siubjects w,ere unable to gen- 
erate the correct proba]t)ilities on the basis of feedback, the^ did 
rB^}ise their estimates "over the course of trials in such -a way 
as to 'correct for conservatism (apparently by. adding a constant) . 

* • jr 

Other investigators hsive al^o/shoV/n that experience in estimating 
posterior probabilities can produce behavior which, if not optimal, 
is more neatly so than before the training began (Edwards, 1967; 
Hoffman & Peterson, 1972;, Southard, Schum, & Bridges, 1964b). 
Such studies establish that^ certain aspects of hypottiesis evalua- / 
tion^ in particular posterior probability estimation, can be im- 
proved somewhat as a result of ^actice. What they do not indicate, 
however, xb how much can be expected of training or how th^ train- 
ing should be done in order to obtain optimal results. 

Another issue that relates to training involves the questic^n 
of how well people can make the p(D|H) judgments that they are ^ 
required to make in some Bayesian systems. It seems to be generally 
assumed t'hat people haye less trouble making these judgments than 
they do making judgments of p^^^|D). In at least one study, how- 
ever, this was not the case,. Bowen, Feehrer, Nicker son, Spooner, 
and Triggs (1971) encountered a fairly strong resistance on, the 
part of experienced military intelligence officers to the idea of 
making judgments of the sort: "If* it is assumed that 'Attack' is 
the enemy commander's course of .action, \what is the probability 
that one will observe the traditional indication ^Massing of Tanks'*?" 
These investigators pointed out that the "generally negative re- 
action to the possibility of estimating probabilities of the type 
that would be required in a Bayesian system must be tempered by 
the fact that the participants were not familiar with the concept 
of Bayesian inference and had not been trained to make the required 
judgments" (p. 103)*. There is, therefore, the question of the 
degree to which training in Bayesian analysis would be effective 
in overcoming the relatively strong preferences that some^ decision 
,maki2rs seem to have for estimating posterior probabilities them- 
selves.^ J ' jm 
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Several other questions cpncerning man's capabilities as they 
apply to hypothesis evaluation have been noted above. These ques- 
tions have arisen because of the results of experijnental studies. 
They are questions which, fori the most part, have not yet been 
adequately answered: The questions, in most cases, suggest some 
'limitation or deficiency in man's hypothesis-evaluation skills. 
To the extent .that these limitations or deficiencies are demon- ^ 
sitrated by further research to be genuine, they represent challenges 
to designers of training programs. If it is the case, for example, 
that probability estimations are sensitive to the format in which 
information is displayed or the mode in which the response is given, 
as sdme studies have indicated, the question is whether such effects 
can be eliminated by training.' If th^y. cannot be, tjien €iie need # 
to be restrictive with respect to display formats and rdspons^ 
modes is so much the greater. Or, to take another example, if the 
way one applies data to the evaluation of a hypothesis is different 
for a favored hypothesis than for an unfavored one, as )ther studies 
have suggested, this constitutes another challenge tq tra-^ning.. 
Can one be trained to apply data to ajl possible hypotheses. in an 
unbiased way Without regard for his preferences? Similar questions 
concetning the potential effectiveness of training can be raised 
concerning each of %he other limitations and deficiencies that . 
have been noted. More research will be required in order to answer 
these questions. 



, ^ 8.7 The Measurement of Suhjective Probability > 

^ Throughout this report we have jna^de frequent reference to 

subjective probabilities, and it has been tacitly assumed that 
•such things can be accurately meas.ured. In fact, how to assure 
accuracy in measurements of this quantity has been a question of 
Some interest. The probl.em is a problem because of the fact that 
the ^probabilities that one obtMns may depend on the way in whi^h 
^they are obtained; or ^ap Toda^ (1963) <puts it, subjective proba- 
bility is essentially defined by the measuring technique that is 
used. Toda /further suggests "feeveral criteria that such a measu-"^ 
ring technique should satisfy: "First, the logical nature" of the 
task presented to the subject should be thordughly understood by 
the experimenter, and, hopefully, by; an intelligent subjept. 
Second, the taSk should involve well-defined payoffs to the subject. 
Third, the task should be so structured that it is to the disad- 
vantage of a subject to respond in a manner inconsistent with his 
expectations. Fourth, since our interest in pleasuring subjective 

^ probability is related to its use J.n>^ decision theory, the measure- 
ment technique shoul,d not be inconsistent with decision theory" (p. 1) 

The third of these criteria is perha|>s the most subtle, and 
has received the greatest amount of attention. Stated in other 
terms, the requirement' is that it be in the subject's best interest 
to state his probability estimates honestl^. That this can be a 
problem may be illustrated by a' simple example of a situation in 
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which the requirement is not m^et. Consider the case of a student 
taking a multiple-choice examination. Suppose he has been in-' 
structed that in answering i^fech question he is to assign a number 
to each of the alternatives associated' with that question in such 
a way as to re'f lect his estimation of the probability l:hat' that 
alternative is the correct one. When he is very pertain of which 
alternative is^corr^ect, all of the numbers except 6ne will be zero; 
when he is less tl)an 100% certain, however, ^e would assign non- 
zero numbers to more than one alternative. Suppose further that 
the score that he^ is to receive for any given question is some 
linear function of. the ratio of the number placed on the 'correct 
-alternative to the sum of the numbers used on all the alternatives 
associated with that question. Given this scoring rule^ the student 
should not distribute numbers in accordance with his true estima- 
tion of the probabilities; instead, he should put zerps on all> 
the alternatives except the one that he considers most , likely, 
^ even if he is not very certain that that alternative is indeed the 
correct one. . > ^ ^ 

\ ' - 

p This IS easily seen by considering a two-alternative case. 

Suppose that the student really thinks that the chances are 7. in 
10 in favor of A being the c62^rect alternative. If he is honest, 
then he 'Will assign 7/10, of whatever points he is going to use, 
on alternative A and 3/<10 on B. Given our scoring rule, and assu- 
' ming tljat our hypothetical student assigns numbers tg "the t;wo 
alternatives in the ra^tio of 7 to 3, then the two yalues that his 
score may assume are 7/10 and 3/10. Moreovet, from the student's 
point of view, the probability of getting a score of 7/10 is 7/10 
(i.<fe., the probability that A is correct), and the probability of 
getting a score of 3/10 is 3/10. Thus, the subjectively expected' 
value of his score is -(7/10)'^ + (3/10)^ = /^8. But suppose that 
our student werfe a gambler, and decided tolput all his;" chances on 
the alternative that he considered most likely to be correct. Now 
the two values that his score ^can assume^ a j:|e 1 and 0, and the Ex- 
pected value of his score (assuming that' he really believes that' 
A's chances are 7 in 10, rather than 10 in 10, as his answer would ^ 
indicate) is 7/10 x 1 + 3/10 x 0 = .70. Thus, whereas the student 
was instructed to assign numbers to alternatives in accordance with 
his judgment of the likelihood* of their being correct, the scoring 
rule is such that he can expect to obtain a higher score by- ignor- 
ing --the i^5structions than by following them. 

A scoring rule that is to satisfy Today's "honesty is the best 
policy" requirement must have what has been referred to as a 
"matching property." In^formal terms, the matching property may 
be stated as follows a Suppose that a subject reports n npn-negative 
values, n ' ' 

r^ r^, . . .r^, Z r. > 0, presumably to reflect the 
J. ^ n 1 > . 

subjective probabilities that he associates ^with alternative 
possibilities', x^^, •••^n* Assume a discrete subjective 
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probability distribution, , p^, ...p , that represents the 



subject's true probability estimates ?:igarding^ 
Letting P, R and X represent, respectivel^^Tthe Vectors ^ ^ 

^Pl' ^2' •••Pp^' ^^1' •••'^n^ -^^if ^2» ***^n^' ^ 

and W(Rf X) the' payoff to the subject, gaven the response vecto?: R 
and' the probability vectOK^ X, the matchifsig property is realized by 
any payoff function for which the following statement is t?rue: . 
The response Vector, maximizes the subjectively expected payoff 
E[W(R, X)l) if and only if R = kP, k being a scalar constant. 
That :is to say, a payoff schema, or a scoring rule, ha^ the matching 
property if and only if the subject max'imizes his subjectively 
J expected paypff when tlie weights that he assigns to the possibilities 
differ from his' true subjective probabilities at. most by the . same 
multiplicative fatftor. N6t^ that when the relationship R = kP 
does hcO-d, tfie ca;Lculation^ of pdds wiri be the same whether based 
on" -R or on^P. - 

Subjective-probability measurement procedures and response 
scoring techniques that make use of functions that have this matching 
prdperty have been referred, to as "admissible probability measures" 
(Shuford^ Albert^ & Massengill, 1966), and "proper scoring" rules 
(Winkler & Murphy, 1968*)*. >^/-Se vera 1 functions with the matching 
property have been defined and investigated, among them the "loga- 
rithmic loss" (Gbbd, 1952; Toda, 1963^, the "quadratic loss" 
(Brier, ^1950; deFinetti', 1962; Toda, ^M63; van Naerssen, 1962, 
and the "spherical gain" (Toda, ,1963; ^oby,* 1964, 1965). ' 

8.7.1 'The Logarithmic Loss Function 

The logarithmic • loss ,f unction is unique a!mong these functions . 
in its exclusive dependence on the value' Off the component of R 
that is assigned to the correct alternative. It is not affec^ted 
by how numbers are distributed over the other components of R, 
The function is given by \ . 

« ^ n 

. W- (R|x.) = k log r. - E r. (35) 
L,-^ a ^ j=l 3 

v;here k is a positive constant, and (^|x.) is read "response 
vector R, given that x^^ is the correct alternative. The subjec- 
\±vb1^ expected p^ysoff, given this functibn, is 



n 



E(W-) = kZEK log r. - Z r. 
which is maxiiai/zed when = Pj^f 



(36) 



' Max E(Wj^) - kZp^ log P^ - 1 (Toda, 1963). .(37) 
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Because. the maximum subjectively expected value is negative — hence 
its designation as a "loss" function — a constaat is often added 
to^ the function in order to make the payoff positive*' Also, because, 
the functidn becomes -«>at r. = 0/ ^ truncated version of it is 
usually employed in practice. ^ / - - 

8.n7.2 The Quadratic Loss Function ' , ' 

The quadratic loss- function is given by " 

■ ^ v*Wo"(R|x.) = Z r,2 + (1 - r.)^ ' ' -^32^ 

^ ^ . kT^i ^ - - ■ . " .. 

when the.nvimber of alternatives is greater than two, ^and by , ' 

W^CEilxi) = -rj^2^ k 7^ i' . . ' • 1 ' (39) ■ 

for ,the two-alternative case; j This function is negative in the' 
two-alternative case (although not necessarily Vhen *the humber of 
alternatives is greater than .two) , sp^r^as in the case of the 
logarithmic loss, a 'constant is often added to the* function to 
assure a positive payoff. ' ^ 

8.7*3 The Spherical-Gain Function , * \ ^ ' 

The spherical-gain function, which has-been elaborated by 
Roby (196S) will be considered in somev/hat more detail, because 
it has some liseful properties that the , other rules/do not 'have, 
and a particularly elegant geometrical representation as wel.to 
The payoff function is given by - * . . 



W^{R|x.) = r. 




{40] 



For a proof that Wg is maximized only when R = kP see Snuford, 
Albert, and Massengill (1966). A reference to tEe example that,, wa.s 
used earlier should be sufficient to make the assertion plausible. 
Consider again the two-alternative examination item for which a 
student thinks the chances are 7 in 10 in ^avor o*f alternative A. 
Recall that if his score is a linear function of the proportion 
of points he assigned to the correct alternative, then nis best 
strategy is to put zero on every alternative except the one he 
considers most likely to be correct; in which case, his (^pacted- 
score would be .70. To 'see that this is not true in .the case of 
the spherieal gain scoring rule, note that if the student puts 
all his stakes, say n points, on alternative A, his expected score^ 
will be: ' 
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7/10 x' {n//n^+0^) + 6/10 x = .70. ' 

If, however, he weights the alternatives in accordance with his 

judgment of what 'the chances really are,, his expected score will 
be: 



7/10 X n/Zl^+S^) + 3/10 X .= .76. 

SI 

It should be noted that the procedure permits the student to 
assign weights to the various alternatives in any way^he sees fit. 
There might appear to be some advantage in forcing the numbers 
assigned to the alternatives for a given item to aAd to one, 
inasmuch as they^ could then be interpreted directly as probability 
estimates. The student could^be instructed to make his assign- 
ments so that they would indeed add to one; however, this is an 
unnecessary demand since the score is unaffected by a change of 
scale. Moreover, if we wish to treat the assignments as proba- 
bility estimates, as we shall in what follows, we can easily 
normalise them by simply dividing each assigned number by the sum 
of the~liumbers associated 'with that question. When this is done, 
and each of the original nximbers is replaced with the resulting 
quotient, then each of the resulting numbers will be referred to 
as a probability estimate, and the collection of numbers associated 
with a given item as a probability vector. 

A nice feature of the spherical gain scoring rule is that it 
provides an easy and intuitively meaningful way of distinguishing 
between one's confidence in the truth of a particular hypothesis 
{or correctness of a test item) and one's general degree of 
"resolution" with respect to th^ overall decision space (or ±o the 
whole test item).- Roby defined, as a "resolution index," 

i2 ,i 



RI = 




(41) 



where RI represents an individual's confidence in his answer.- 
Equation 41 is simply the denominator of equation 40 after the 
latter has been normalized.. ^ * ~ 



As in the case of W , the maximum value of RI is 1. It should 
be clear that RI = 1 only if rj = 1 for one value of j arid Of or 
all others. That is to say, in keeping with our intuitive notions 
about how an index of confidence should behave, it assumes its 
maximum value when one has put all his chances on a single alter- 
native. (Note that whether that alternative is correct or 
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incorrect is irrelevant to this measure-^as it should be.) Unlike 
Wg, Rl canrjt assume the value 0- Its minimum value depends on*the 
number of alternative hypotheses under consideration, or — in the 
case of the examination example — the number of candidate answers 
associated with a question. It is obtained when . 1 

""j " Sr. 

J 

for all j; that is, the index gets its lowest ^alue when the same 
number is assigned to every alterriative. Agaih, this is consistent 
with our intuitive ideas about confidence. The fact that the 
minimum value of the index depends on the number of alternatives is 
also in keeping witK our intuitions about how a^ measure of confidence 
should behave: one should have less conf idence* in a guess^ among 
three equally likely alternatives than in a guess, betwe'enxtwo of ' 
them. 

• « 
8.7.4 Implementation of Admissible?* Probability Measures 

One of the practical difficulties in applying scoring rules 
with the matching property is that of providing subjects with 
intuitively meaningful information concerning the implications of 
their probability assignments vis-a-vis the scores that cpuld re- 
sult from them. It is clear that simply providing individuals with 
formal expressions of the rules will not suffice, at least for* 
those who are not matnematically trained • One approach to this 
problem' is that of illustrating the implications of any given rule ^ 
wit^ concrete examples that make clear the advantages of being 
honest. Another, and perhaps prfeferred^ approach is. that of pro- 
viding the individual with an explicit representation of th^ payoff 
that he would receive, given the truth of any specific hypothesis 
and t^e way in which he had distributed probabilities over the 

alternatives. 

■> 

"^Organist &n4 Shuford designed ^a pap^r and pencil procedure 
for providing this' information in the case of the logarithmic 
loss function (Baker, 1964; Organist, 1964; Organist & Shuford, 
1964). Shuford (1967) and Baker (1968) have also described a 
computfer-'based technique for providing similar -^information in a 
dynamic way. '^In this case the alternatives open to the decision 
maker are shown on a computer-dr'iven display. Associated with 
each alternative is a line, thfe length of which represents the 
user's relative confi^ience that that alternative is the correct 
one. The user adjusts the lengths of the lines by means of a 
light pen. When the length of one line is changed by the user, 
the lengths of all the others. are adjusted 'by the computer so as 
to constrain the sum of the lengths to add to one^ at all times. 
Also displayed with each line is a number which indicates to the 
user what his payoff would be if the alternative associated with 
that line were the correct one.. The loga^-ithmic loss function ' - 
determined the ^relationship between the number Representing 
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potelitial payoffs and the lengths of the lines in the applications 
of the system that are reported • However, the relationship could 
as well have been determined by any other scoring, rule of interest. 

8.7«5 The Efficacy of Admissible Probability Measures 

It would seem clear from the mathematics of the , situation * 
that scoring rules that have the matching property should be ysed 
in preference to those that do not. It has not been clearly de- 
monstrated empirically, however, that subjec^ts* tend *to behave 
dishonestly if such rules are not used, or that their responses 
are free of biases if they ,are (Aczel & Pfanzagl, 1966; Jensen & 
Peterson, 1^973; Samet, 1971; Schum, Goldstein, Howell, & Southard, 
1967) . Moreover, it is also apparent that many of the probability 
estimation situations of interest to investigators of decision 
making are situations in which the only scoring rules that are 
operative axe.J;h|^^ imposed by nature. The s. tuations 

in which ^bjectij;^ probabilities are of greatest p;racti^al sig- 
nificance tenas^to be those in which the payoffs are beyond the 
experimenter's control. 

8.7.6 Subjective Probability Measurement and Training 

One question of interest that relates to 'training research 
is whether individuals who have had experience' in making ^probability 
judgments in controlled situations with scoring rules that have 
the matching property are more effective at judging probabilities 
in real-world situations than those who have had experience at 
estimating probabilities .but have not been exposed to matching- 
property rules. As has already been noted, some investigators 
have advocated the use of expert^ to estimate conditional proba- 
bilities to be used in Bayesian aggregation systems (Bond & Rigney, 
1966; Edwards, 1965b). Often, however, it is not possible to 
determine how accurately such estimates are ipaiie. If one had an 
objective indicant of the probabilities of interest that was in(;Je-" 
pendent of the experts' judgments, it would not be necessary to 
get the' judgments. It would be .of interest, however, to determine 
whether the behavior of experts on such tasks would be sensitive 
to the type of experience they had had in estimating probabilities 
in controlled situations, and in particular to their exposure to 
admissible or inadmissible probability measurement techniques. 
Savage (1971) has- suggested the early introduction of admissible 
scoring rules to* children, albng with careful training in the 
assessment of opinion strength, could have the salutory effect of 
dispelling some of the myths concerning the relationships fcjetween 
certainty, belief and action— -e.g. , the idea that one should speak 
and act as though certain, even when one is not^ and the notion 
that weakly held opinions are worthless — that are fostered by 
conventional educational testing methods. 
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Scoring rules With the matching property answer to one aspect 
of the problem of measuring subjective prpbabilities^ ^namely that 
of structuring the situation so that honesty in reporting is the 
best policy* There; are other aspects of ^he problem, ! however , that * 
are not so readily solved. Expressions of certitude have been shown 
to vary considerabl:|r as a function of. the way in which they are 
^ reported (Samet, 19:71) and of the'cphtext in"* which they are obtained 
(Nickerson & McGoldJrick, 1963). Typically, when subjects arfe asked 
to rate their confidence in their^ own performance on a perceptual * 
or cognitive task, a positive coirrelation between these variables 
is found — confidence is highest when performance is best — but the 
strength of the ^Relationship i,s not always great, a^d the signifi- 
cance of -a givexi confidence rating depends on the situation an4 the 
person making it (Andrews & JRingel, 1964; Nickerson/& McGoldrick,- 
1965) • A fundamental question that is raised by these results is 
whether such /Factors affect certitude itself, or only its expression. 
A further q^ifestion is whether such variability— rwh^^te^'er its basis — 
can be eliminated, or dt»least significantly reduced, as a result 
of appropriate training. " 

8.8 The Use of "Unreliable Data 

In the foregoing discussions of the use of Bay es rule, it has 
been tacitly assumed that the data used in estimating conditional 
or posterior probabilities had been accurately obsierved and re- 
ported. In the chips-in-urn illustrations, for exiample, it was 
assumed that' one could examine a chip and determine Its color easily, 
or that someone else determined the color and reported it accurately. 
Thus, the decision maker could operate with complete confidence in 
thQ data at Ms disposal. In the real world of decision mak.ing, 
things often are not this way. Frequently, the denervation or the 
reporting of events is faulty, and the decision maker is obliged 
, to take this fact into account when making use of the data that he 
' has obtained. 

We naturally assume that data from a trustworthy source will 
be more useful to a decision maker than will data from a source 
that has not inspired confidence in the past. The use of an 
explicit reliability rating procedure for intelligence reports by . 
NATO army forces (see Section 5.2) is based on such an assumption. 
Few attempts have been made, however, either to validate this 
assumption or to determine in a quantitative way exactly how con- 
fidence in a data source does affect the way in which the data 
from that source are applied to a decision p^fpblem* 

8.8.1 Prescriptive Approaches 

One class of pre^scriptive models for taking into account the 
reliability of data has come to be known as "cascaded" or "multi- 
stage" infer.enee, suggesting a process of hypothesis evaluation 
• that involves more than one. step. Schum and DuCharme (1971) point 
out that research on cascaded inferejice has been focused on two 
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sit^uations: one, in which the observer or reporter of an event ex- 
his degree of certainty concerning whether or not the event 
^^tT ll ^^^^^^^'^ (^.es' example, Dodson, 1961; Gettys & Wilike ? 

1959; Steiger & Gettys, 1972), and a second, in which the report of 
an event IS made without qualification by a source that is known ?o 
??7^^^c u^^"" perfectly reliable (see, for example, ScJium & Sarme 
1971; Schum, DuCharme, & DePitts, 1973:- Snapper s Fryback, 1971K 

In both cases, attention has been confined primarily to 
relatively simple situations, e.g., those in which (1) the decision 
maker's task is to determine which of two hypotheses, H, or H„, is 
true, and observations have only two possible outcomes, and D_, 
and (2) the reliability of a report is independent of the 
hypotheses that are being considered, that is to say, event 
and D2 are neither more nor less likely to be confused under 
than under 

^^^^il considered the situation in which an observer ' 
IS not certain which of two mutually exclusive events, D, and D 
has occurred, but may be able to make a probability or cL^itudl' 

Jhf^nn^J °- ^K^K^^?^- ^'^ suggested that in order to cJicSlate 

the posterior probability of a hypothesis in this case, one should 

?akrfwSiaSLr^"'' possible^ events ani ?hen 

bi^f t?p^ ih^^ ^v, ""^^^ ^^^^^ values, the weights being the proba- 
bilities that the observer attaches to the event possibilities 
aslolSis possible data, the calculation may be represented 



■■ C(»i|D) = V»(D^)p(H..iD^) + iP(D2,)p(H. ID2) ' (42) 

C(H^jD) is the posterior probability of H.,. taking the ob- 
lh^J%li uncertainty into account, and ^(D ) isHhe probability 
l^!n^- S °^^erver attaches to the possibility that he has observed 
fTnn\^j; More generally, given n possible events and the assump- 
Srf^±i^^:.°^r;rL??L!^^!°^ ^ probability-^ each of them,"^ ' 



where 
server 



the fonnula might be written as 
?(hJd) = 



Substituting the Bayesian formula for p(Hj^|Dj) we hav6 

p(Dj|Hj^)p(Hj^) 



;(H.jD) = ZiMD.) = 3 7^ i_ 

1' j'^^ 3' ^ P(D^|H.)p(H.) 



(44) 



no^o? f Dodson 's work as a point of departure, Gettys and Wilike 
(1969) and Schum and DuCharme (1971) gave the process of dealing 
with unreliable data a more explicit two-stage form. The following 
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discussion roughly follows Schum and DuCharme. These wrdters fo- 
cused on the case in which a dec ision maker obtains inforination 
about data via a source t^at sometimes incorrectly reports what 
has actually occurred. (It is irrelevant to this discussion whether 
the source's errors are assumed to be errors of observation or 
errors of report,) For each of the decision problems that wer4 
analyzed there were two hypotheses,, and H^f two possible data 
events, D. and D^/ ahd two possible reports By the source, d. and d... 
What one Wants to -determine is p(H^|dj),* - ^ 

According to Bayes rule 



p(Hi|d.) = 



P(-dj|H.)p(H..) 
P(dj) 



(45) 



The problem then is to determine p(d. |H.). If the pi^v bability of 
a datum conditional on a hypothesis, -^p (6, |H. ) , and the ^probability 
of a report, conditional jointly on a hypot^iesis and a datum 
(p(dj |h. flDj^) , are known, theii the probability of a report, -condi- 
tional 6n a hypothesis p(d. |h.) can be easily calculated. The 
relationship-- is given by- ^ 



p(dj|H^) = Zp(Dj^|n^)p(d. iHilDj^), ' 



(46) 



a graphical representation* of which is shown in figure 26 • When, 
by assumption, the reliability of a report is independent of the 
hypothesis that is being considered. 



p(djlH.nDj^) = p(djlDj^) 
so, in effect. 



(-47) 



p(d.|H.) =-£p(Dj^|H^)p(dj|Dj^) , 



(48) 



Schum and DuCharme refer to 



A = 



_ p(d.|H.) 
" p(djlHj^) 



(49) 



ERIC 



*Our notation differs slightly from that used by Schum and DuCharme: 
we use and D2 to represent the two possible data events, whereas 

they used D and 6, and we__use d^ and dj to represent reported data 
whereas they used D* and 5*. 
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Figure 26, 

r 



Graphical representation of derivation of 
p(d.|H. ) and' adjusted likelihqod ratios 
for less than comfJletely reliable data 
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* as the "adjusted likelihpod ratio," the likelihood ratio that takes* 
into account the 'degree of reliability of the source. Of course,, 
A reduces to the standard likelihood ratio when the source is 
assumed to give completely reliable reports, inasmuch as, -in this 
case 

The way to deal with the problem of unreliable data , then, accor-^ . 
ding, to Schum and DuCi^ayme is with a two-step process: (1) adjust 
the diagnosticity of the data by determining p(d. |d, ) or A, and 
(2) apply the adjusted data to revise the distribution of pro- 
babilities over the hypotheses via Bayes rule. 

What one must be able to measure (lir esji-imate in order to 
use' this procedure .are p(DiH), the standard condition il probabilities 
of Bayes theorem, and p(d|D), the indices of source reliability ♦ 
Schum and DuGharme define source reliability in terms of - ^ 

r = p(di|D.) ' ,. ^ ■ ^ . . . 

the probability that the source will report a data event accurately/ 
They distinguish four different decision "cases" in terms of cer- 
tain symmetries and asymmetries involving p(d|H)'and p(d|D), and 
they develop the implications of their prescription for dealing 
with unreliability for each case. The cases that they distinguish 
are: 

Case I: Symmetric p(D|H): • Symmetric p(d|D) 

^P(D^Ih^) = p(D2|H2); p(d^|D^) = p(d2|D2); 

Symmetric p(d|D) 
p(d^|D^)' = p(d2iD2). 



1 for j ~ k 
0 for j 7^ k. 



Case ri: AsiTmnetric p^|h) ; 

\ P(D^|h^) (D2jH2') 



Case III: Symmetric p(d|h): Asymmetric p(d|D) 

PCD^IHj^) =p(D2|H2); p(d^|D^) ^p(d2|D2)- 

Case I^: Asymmetric p(d|h); Asymmetric p(d|D) ^.'j 

.pCDj^lF^) ^ p(b2|H2) ; p(d^|D^) ?t.p(d2|D2). 
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In order 'to avoid the use of conditional probability nota- 
tion, Schum and DuCharme introduced the following notational 
equivalencies: 



For symmetric p(d|h) 



For symmetric p(d|D) : 



For asymmetric p(d|D) 



p = p(D^ 
1-P = .P(D. 



For asymmetric p(d|h): 





i p(Di 






= P(D^. 


H2) 




= P(D2 




I-P2 


= P(D2 


H2) . 



r = p(d^ 
1-r B p(dj 



H.) 



D.) 



^1 




Id,) ■ 




^2 


E p(d2 


D2) 




1-r, 


= P(d2 








= p(d. 


°2)- 





* Let'ting the subscripts on A represent symmetry or asymmetry 
with respect to p(d|h) and p(d|D), respectively, and malcing the 
above" substitutions into equation (46), as appropriate, we obtain 
Schum and DuCharme 's expressions for the prescribed use of data 
of imperfect, but known, reliability for each of the four cases 
they considered-^ All adjusted likelihood ratios represent 



p(d^|H^) 
PTH^TIV 
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's,s • {l-p)^+p(i-r) 



(50) 



Case II: 



or, equivalently. 



a, s 



a, s 



p^r-Kl-^) (1-r) 
P2r+(l-p2M.l-r) 



p^+k 



(51) 



(52) 



where 



^ ~ 2r-l' ^^'^ ' 



(53) 



Case III: A. 



_ pir^i + d-p) (l-r2) 
~ (l-p)rj^+p(l-r2) 



(54) 



or if pj^l and r27^1. 



where 



s.a 



(55) 



c = -^1 



1-r. 



(56) 



Case IV: 



^ P]^r^+(1-P^) (l-r2) 



(57) 



or if r^7^(l-r2) 



p,+b 

[ = _± 

a, a P2+^ 



(58) 



where 



1---, 



ri-d-r.,) 



(59) 
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It follows from the definitions of unadjusted and' adjusted - 
likelihood ratio that the latter is always closer to unity than 
the former and that the difference between them increases as 
reliability, r, is decreased from 1.0 to 0.5* (except when the ♦ 
data, are completely uninf ormative to begin with and the unadjusted 
ratio is 1) . This is consistient wi.th the intuitively compelling 
requirement that the less reliable the data, the less diagnostic 
impact it should have.** Figure 27 shows how the difference be- 
tween unadjusted and adjusted likelihood ratio grows. as reliability 
is decreased, and how the adjusted ratio goes to 1 for r = .5, for 
the case in which both p(D|H) and r are symmetri-c, i.e., Schum and'. 
DuCharme's Cage I. ^ 

Figure 27 also illustrates the fact that the greater the 
diagnostic impacj: of data (when reported by a completely reliable 
source) , the greater is the effect of a decrease in reliability of 
a report. This also is an intuitively reasonable relal - onship: 
the less informative data are to begin with, the less there is to 
lose if they are reported unreliably, what is less intuitively 
apparent is the fact that even a very small decrease in reliability 
may have an extremely large eftect on. likelihood ratio if the un- 
adjusted ratio is very high. Schum and DuCharme (1971) point out, 
for example, that in Case I, if a datum with an unadjusted likeli- 
hood ratfo of 100,000 is-reported by a source with a reliability 
of .99, the adjusted ratio is reduced by about four orders of 
magnitude to slightly less than .99. 

The results of Schum and DuCharme 's analysis bear on issues 
relating to the design of information and decision-making systems 
and on the role of humans therein. For example, they show that under 
Case I conditions, there is a reasohably straightforward tradeoff 



♦Decreasing r below 0... 5 has the effect c^f making the adjusted 
. likelihood -ratio depart again from unity,- although it still remains 
closer to unity than does the unadjusted ratio. In other^ords 
decreasing the reliability quotient below 0.5 increases Sidiai- 
nosticity of the data, but in sunport of the altiHiti^^ hypothesis 
This IS c5onsistent with the idea that a source that is consisSn^^y* 
wrong mgy be very informative; one ne^d only interpret its report 
as evidence of the opposite of what it says, m this, discussion; 
we will confine our atte^ition to the case in which 1.0. > r > 0.5.-' 
**Schum and DuCharme (1971) point out, however, that when tHe 
reliability of report is not independent pf which hypothesis-is 
being considered, it is possible for A to differ more from 1 Chan 
does L; that is to say, it is possible for a decrease in relia- 
bility, m that case, to increase the diagnosticity of data. ' 
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Figure 27. Adjusted likelihood ratio (A) is a function 
of data reliability (r) for several values 
of unadjusted -likelihood ratiovPL) , for Schum 
and DuCharme's Case I > 
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between p(d|H) and. p(d|p). And' the tradeoff is such that if one 
wantg to increase the diagnostic^ impact of information flowing , 
through -.a \^-y St eir.v and /the costs of increasing jthe conditionals 
P(d|h) and^^p(d|D) are^^ equal, one should increase the smaller of 
the two. . 1 , 

^ Also, the analyses show that in Cases II and IV A is dependent 
upon specific values of p. and p^ rather than on 'feheir ratio* Thus, 
despite the fact that earlier results have suggested that people \ 
find* it easier to make judgments of likelihood ratios than of \ 
conditional probabilities, there may be situations in which esti- 
m'ites of the lat-feer should beurequired. 



8^8.2 Some Empirical Results 

The models developed, by Schum and DuCharme are prescriptive, 
providing for optimal adjustment of the likelihood rativ. under 
conditions^ in which .data are reported with les^than total, hut 
knowA, reliability. We now* turn to a consideration of several ^ 
studies aimed at com{)arin?^ actual performance against that prescribed 
by these model? • In the next section we then present a ^brief 
account of some descriptive models suggested by these results. 
^All experiments and models that will he considered ^n these section^ 
^address situations wheire input to tfie decision process is an event 
or set of events reported by^ a single unreliable source. 

Snapper ^nd Fryback (1971) present ,the results of a study in 
which tlJe experimenter reported to the subject with (symmetric) 
reliabilities Qf 1.0, 0.9, and 0.7 the outcomes of events concep- 
tually similar to the draws of chips from an urn. The probabilities 
of events conditi9nal on hypotheses, p(D2^|H^), p(D2|H^) and 
P(D*|'H^), p(D^|H;^j, were, respectively, as rollows: ia)0.33, 0.67 
and'^0.57, 0.33; tb) 0.80, O^^^O and 0.60, 0.40; (c) 0.90, 'o. 10 and 
0.45, 0.55; (d) 0,.25, 0.75 -a'nd 0.75, 0.25. For conditions in ^ 
which the experimenter's reliability was equal to unity, only (a) 
and (b) Were used. Subjects were required to indicate which of ^ 
the hypotheses they considered miore^likely as a result of the ex- 
perimenter's report, and how much more like'ly than the alternative 
hypothesis they considered it to be. Under condiliions of unit 
reliability, subjects' estimates corresponded very closely to the 
actual likelihdod ratio, but when reliability was less than unity 
they represented slight underestimates of the impact of the least ^ 
diagnostic reports and overestimates of the impact of the remaining 
reports. The extent; of this overestimation, moreover, increased 
with the mag^nitude of A. 



Johnson (1974; see 'alst> Johnson, yCavanagh, Spooner, & Samet, 
1973) ^has'utilized a similar task and/response structure to 
study the effects of four different variables on cascaded ' ' 
inference: (1) s amplg size , the number of draws which underlay 
axcumulative outcome report (e'.gl, "three reds, two blacks"); 
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(2) data generator dia^nosticity , the relative composition of red 
and black chips m the^ urn; (3) sample diagnosticity , the diagnos- 
tic value* defined by the difference between total numbers of red 
and of black chips underlying a report; and (4) source reliability . 
Posterior-odds estimates that were obtained in this case were found 
to be sensitive to different values of sample size, data generator 
diagnosticity and source reliability, tending to decrease as the 
values of these variables decreased. When the report was known, 
to be perfectly reliable, estimates of posterior o.ids v/ere generally 
more conservative than those computed from Bayes^ theorem; however, . 
they became progressively less conservative and approached optimal 
values at intermediate levels of reliability (•8-.7), and' then 
became slightly excessive at lower levels {•7-,.6). 

The diagnostic^y and reliability of reported events were 
manipulated by Youseff and Peterson (1973) in such a way that the 
value of A in a situation requiring multistage infert^nce was 
equal to the value of the standard likelihood ratio in a single- 
stage situation (that is, one with report reliability !equal. to . 
unity). Subject"^' estimates, tended to be conservative for high 
values, both of ft" and cf L, as compared with the Bayesian mbdel, 
and tended to be excessive. for low values* The odds estimated 
in conditions requiring multistage inference were consistently 
greater than those estimated in singl<e-stage conditipns and, as 
a result, were excessive compared to the optimal odd's over a i 
wider range than were single-stage odds* t 

Schum, DuCharme, and DePitts (1971) conducted a study in whicl\ 
the accuracy of subjects^ own . observations of ^the number of Xs 
contained in tachistoscopically presented 4x4 matrices of Xs 
and Os constituted the reliability levels. Subjects were required 
to estimate the relative likelihood of twb possible hypotheses 
renting to the data generator after e^ch of five stimulus ^ 
ptesentations.' Ui^der conditions in which sufficient time was 
available for totally accurate observation of the stimuli, esti- 
mates became increasingly conservative compared to the optimal ^ 
model as the diagnosticity of each observed event -and the infe- 
rential consistency over a set of five events increased. Under 
conditions in which insufficient time was availabl^e for accurate 
observations, the subjects' estimates were generally close to 
optimal or slightly excessive when diagnosticity and consistency 
were high, and became more conservative as eithe^r of these para- 
meters assumed lower values. In k second phase of this same studi>, 
subjects estimate^ directly the diagnosticity of data based on 
brief obsfervations of each slide. Compared with the optimal model, 
such estimates become increasingly exgessive as L increased. 

The results of these studies establish that the behavior of 
decision makers is indeed influenced by the degree of reliability, 
of their data sources. They also jidemonstrate, however, that 
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performance tends not to%be consistent witb that prescribed by 
the formally appropriate rule for adjusting data diagnosticity. 
Further, performance with unreliable data often differs in one 
important respect from that that has been observed in classical, 
Bayesian irnference situations in which events are observed, or 
reported, ^Ith perfect accuracy.. Whereas in the latter case the 
decision maker's estimates, though revised in the appropriate di- 
rectiprf/ tend to be conservative as compared with Bayes theorem, 
his estimates based on less--than-completely-reliable data fre- 
quently appear, to be excessive as compared with Schum and Du- 
Charme's prescription for optimality. Because the value of A as 
defined by the Schum and DuCharme model, in effect, makes an adjust 
ment in the direction of increasing conservatism (produces a value 
closer to unity) , the two effects^ — conservatism vis-a-vis L and 
excessiveness vis-a-vis A — cah offset each other, if conditions 
are just right. 

8.8.3 Some Attempts to Develop Descriptive Mo dels of 
Cascaded Inference ^ 



As we have noted, the model developed by Schum and DuCharme 
(1971) for dealing with unreliable data prescribes two steps, or 
stages: in the first stag'e, the nominal diagnosticity of a datum 
is discoiinted to reflect the degree of reliability of the source, 
and in the second, the adjusted datum is applied to the hypotheses' 
under evaluation in accordance with Bayes rule. If hypotheses 
are being evaluated in terms of odds, the process can be repre- 



sented as follows: 

Stage 1: compute A = 



p{cl|H^) 
P(d|Hj) 



Stage 2: compute Af?Q 
where A represents the adjusted likelihood ratio, and and Q 



represent the posterior and prior odds, respectively, 



1 ^'0 



The experimental results that were reviewed briefly above 
make it clear that people typically do not behave in accordance 
with this prescription. Several investigators have attempted to 
develop models that do describe behavior. 
I 

The results obtained by Snapper and Fryback (1971) , using 
symmetric reliabilities, suggest that in dealing with unreliable 
data, decision makers estimate the likelihood ratio as though the 
data were completely reliable, adjust the resulting ratio by mul- 
tiplying it by the reliability quotient, and then apply the ad- 
justed ratio to the calculat.«on of posterior odds. The process 
may be represented as follows: 
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Stage 1: compute X = rL * ' ' 

V 

§i;age 2: compute = 
• X u 

o?^Kf ^^""'^ fryback note that the optimal rule for the f{rst stage ' 
. LE^o^r neither apparent nor intuitive, whereas the rule 

that seemed to describe the behavior of their .subjects has some 
xntuitive appeal and is easily applied, its use leads, howeve?? to 
subjectxve estimates of livelihood ratio that are ^xc4ssive in 
' compa-rison with those prescribed by A. That if ?o say'T Lads to 

" °^ ""^^ diagnostic impac't of a given (uri^eliabSf 

• case ?-^if ^??nJS f^""^ ^overestiipates A~for Schum and Ducharme's 
both f i ^i'Sures 28 and 29. Figure 28 shows 

both A and A as functions of , r for several values of L; figure". 
onl5 ? t° same. conditions. The figurls show ' 

sam. Zitl \ u 1 ^^'^ ^ ^ .5. For L < 1, 6ne obtains the 

same relationships by sin-.ply expressing the lilcelihobd ratio H. 
5^1 rather than H, re If The case of r <.5 is of little' ^ 
interest for the reason explained in the first footnote on page 134. 
As may be seen from these figures, ■ the. degree to which ?( over- 
estimates A depends both on L and t: for given L it tends to 

t 

. 9®ttys/ Kelly, and Peterson (1973) have suggested a model " 
that IS slightly 'diffe'ient from that^of Snapper and Fryback. It 
assumes that the decision maker estimates posterior * odds on the 
assiimption that the most likely event is true, andv^thiiTadjusts 
V I ? reflect the reliabili,ty of the data sborce. This 

model may be represented as follows: 

Stage 1: compute ^.^ = lQ^ ^ 
Stag6 2: compute ti^ = rfi^. f 

It is apparent that although the process by which the pos- 
terior odds are estimated differs in the two cases, the results 
are precisely the same. Edwards and Phillips (1966) have presented 
evidence, however, suggesting that the way in which people estimate 
posterior odds may be better described by ; ' 

n _ C ' • ^ 

"l - ^ ^0' (60) 

•where c varies with L, than by the prescribed U. = lQ^. punaro 
(1974) points out that the models of Snapper and FrybSck, and of 
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Gettysr Kelly, and Peterson make different predictions if the odds 
are calculated according to Phillips and Edwards' expression. The 
former leads to ^ 



(l^ = (rL)^J>Q (61) 
and the latter to ^ . 

ti^ = riPsiQ ' ■* ^ ^ ' (62) 

Funaro (1974) has recently attempted to evaluate the pre- 
dictive power of Snapper and Fryback^s model and of that of Gettys, 
Kelly, and^^Peterson, using both L and*L^ as unadjusted likelihood 
, ratios in each case. A symmetric p(D|H) — symmetric "r task (Schum 
and DuCharme, Case I) was used. Subjects were require>. t6 revise 
odds' estimates under both single-stage (perfect-source laliability 
assumed) and cascaded-inf erence conditions. Values of c were 
estimated separately for individual subjects from data obtained 
in the single-stage conditions. 
^ m . ' ^ 

The results were not consistent with any of the models de- 
sj::ribed above. They were predicted best by another model that 
Funaro proposed. This model, which Funaro called the empirical 
model, assumes that subjects accurately estimate A, and then apply 
this estimate to the revision pf odds with the same degree of ef- 
f ect^iveness, or ineffectiveness, with v/hich they apply L in single- 
stage tasks. The conclusion appears to be inconsistent with the 
results of Youssef and Peterson (1973) who found that odds's es- 
timates made under cascaded conditions were consistently excessive 
relative to thOjSe made in single-stage tasks, given A = L. 
Funaro notes, however, that subjects in his experiment could 
have acquired a direct appreciation for A from the proportion 
of successes and- failures in a series of reports obtained 
from the source during the course of the experiment. (In a sym- 
metrical p(D|H) chips--dn— urn situation, one can unambiguously 
define a "success" as the drawing — or in this case reporting — of 
a chip of the predominant color.) To the extent that subjects 
were able to develop a direct awareness of A, the effect would 
have been to eliminate the need tor a two-stage process and to 
transform the task into the simpler problem of revising odds on 
the basis of totally reliable data. The suggestion is an eminently 
plausible one and the possibility that this < is in 'fact the way 
unreliable data are often accommodated in real-world situations 
deserves further study. 

* 8 . 9 Some Comments on Bayesian Hypothesis Evaluation 

inasmuch as the Bayesian approach to hypothesis evaluation 
has repeived so much attention by decision theorists ^nd investi- 
gators of decision making, it' seems important to consider some of 
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the limitations of th;Ls approach. To point out limitations is not 
of course, to deny that the approach has merit. Among its advantages 

are the fact that it places mj.nima]^ demand memor_y_J3ecause -data 

can M discarded after being used to" update the distribution of 
probabilities over hypotheses, the fact, that it provides a means 
of aggregating qualitatively^ different data in a meaningful way and 
the fact that the procedure for applying data to the evaluation of 
hypotheses automatically -weights data in terms of their relevance 
to the hypotheses being evaluated. It is precisely because the ap- 
proach does work well in some contexts that there is a danger of 
uncritically concluding .that it is appropriate in all cases. The 
following observations are based largely on a discussion by Bowen 
Nicker son, Spooner, ,ahd Triggs (1970) . ' 

.7' . . 

First,^ Bayes rule itself applies to only one of the several t 
aspects of decision making,- namely, hypothesis evaluation or, more 
precisely, -the resolution of uncertainty concerning tue state of 
the world. Whatever its efficacy Tor that particular task, it is 
not the grand solution to the problem of decision making. 

« 

Second, application of Bayes rule requires that the decision 
problem be structured in a very precise way. In particular, it 
requires that one's uncertainty about the state of the world be 
represented as an exhaustive set of mutually exclusive possibilities. 
It does not, however, provide any help in identifying these possi- 



bilities. 



Third, the requirement for an exhaustive set of mutually ex- 
clusive hypotheses "about the state of the world precludes the 
possibility of expanding. one ' s hypothesis space as one goes along. 
It clearly often is the case, in real-life situations, that new ^ 
hypotheses are suggested by incoming data. That is to say, obser- 
vations may have the effect not only of modifying the credibility 
of existing hypotheses, but of suggesting new hypotheses as well. ' 
. ' ' ' 

, Fourth, the fact- that use of Bayes r\ile presupposes a set of 
mutually exclusive hypotheses has another implication. By defini- 
tion, one and only one of these hypotheses can be true; all the 
others must be false. .The probabilities thatt are associated with 
these hypotheses do not, of course, represent their truth values, 
but, rather, the decision maker's opinion concerning their truth 
or falsity. It was pointed out in the preceding paragraph that no ' 
provision is made for the possibility that tl\e hypothesis set does 
not contain the true hypothesis. it is also the case that provision 
IS not made fpr the possibility that more than one of the hypo- 
theses are .true, or that one or^more is partially true. 

Fifth, application of Bayes rule is a recursive process: each 
time that a new observation is to be used to update a posterior 
probability estimate, the posterior probability from the preceding 
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^update is used as the prior probability for the current update. 
Originally, hov/eyer — before the first observation is made— the 
prio^; probabilities must be estimated, and Bayes rule does not 
Belp in tliis regard. Investigators are not entirely agreed on how 
these prior probabilities should be assigned — or on what they mean. 
It is often pointed out that how prior probabilities are assigned 
may make little difference (provided values very close to 0 or 1 
are not used) , because the effect of the initial values will be 
largely nulled after several observations have been made. The 
problem can be a significant one, however, when hypotheses ifiust be 
evaluated on the basis of relatively few data. In such cases, the 
initial prior probabilities can have a very strong effect on the 
final posteriors, and thus the way in which they are assigned is 
of considerable concern. 

Sixth, the basic assumption that justifies a Bayesian approach 
ts> hypothesis evaluation is^ the assumption that man is better at 
estimating p(D|H) than at estimating *p (H | D) . We have noted in 
preceding sections some experimental evidence that tends to support 
this assumption. We have also noted some studies, however, that 
have shown that thi^ result is not always found. Moreover, th^re 
is a question concerning how far the evidence tfiat does support 
this assumption can be pushed. The only way thkt one can determine 
how accurately a man can estimate p(d1h) is to observe his perforr 
mance in experimeiital situations in which p(D|B^) is objectively 
defined or can be determined empirically. But, typically, in real- 
life situations of greatest interest, p(d|h) is not known, and 
cannot be determined empirically--which is why is must be 
defined qr can be de1:exinined empirically — which is why it must be 
estimated. The question arises then, if it is not known, how can 
we be sure that one*s estibate of it is accura4:e? And the answer 
is that we cannot. How much confidence one should have in the. Con- 
clusion that man is better at estimating p(d|h) than at estimating 
p(h|d) in real-worJLd situations depends in large part on the extent 
to which one is w;Llling to assume that what is known about perfor- 
mance in laboratory situations in which p(d|h) usually has a 
straightforwaird r^lative^-frequency interpretation, is generalizable 
to real-world situations in which it does not. 

Seventh, Bayes rule does not provide the decision maker wi;th 
a criterion concerning when to stop processing incoming data and 
to make a decision. Inasmuch as data gathering can be costly in 
terms of both time and money, it is essential that any completely 
adequate prescriptive model of decision making hav.e an explicit 
stopping rule to indicate ^hen hypothesis evaluation should be ter- 
minated and a decision made". 
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We emphasize that these conunents deal with limitations of 
Bayes rule. One might argue that the observations are unnecessary, 
on the grounds that proponents of Bayesian diagnosis have never 
claimed that these limitations dp not exist. It seems to us 
important to make these limitations explicit, however, because 
they help to place the'notion of Bayesian decision making in 
perspective. The idea of obtaining estimates of p(D|H)xor of 
likelihood ratios from humans and using \hese estimates ^.o update 
posterior probability, distributions 'in accordance with Bayes theorem 
is undoubtedly a reasonable approach to evaluation in some situa- 
tions. It is not always appropriate or practicable, however, as 
sdme^'Bayesians have been careful to point out. Edwards (1967) de- 
scribes the situations for which the approach is most appropriate 
as those that have one or more of the following three characteris- 
tics: "the input information is, fallible, or the relc^tion'of input 
information to output diagnostic categories is ambiguou3 or uncertain 
or the output is required to be* in explicitly probabilistic form" 
(p. 71). ' ^ 
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SECTION IX 

PREFERENCE SPECIFICATION 

It is generally assumed that a decision maker is not indif- 
ferent to which of the various jpps_sJ^ble.decision.„Q^^ _ 
Us we ha'O-e , already noted, in some formal representations of de- 
cision situations, the decision maker's perferences with respect 
to the possible outcomes are made explicit in a payoff matrix. 
The contents of a cell of such a matrix is the worth to the de- 
cision maker of the choice of a specific action-alternative, given 
the truth of a specific hypothesis concerning the state of thfe 
world. The entire matrix presumably' represents the situation 'V 
fully: It identifies all the decision maker's actipn alternatives 
as w^ll as all the possible states of the world, aM shows for each 
aiternative-state combination its worth to the decision maker. 

A Difficult and Peculiarly Human Task • ' 

' The problem is how to determine these worths. There are two 
observations to make in this regard. , The first is that this task, 
more than any other associated v^ith decision making, is peculiarly 
human. One would expect that many of the decision-related tasks 
that now must be performed by humans will in time be performed by 
computers. However, the specification of preferences for decision 
outcomes ihvolves value judgments. To say that one decision out- 
come IS better than, worth more than, or preferred to, another is 
to say that it represents a greater good within the context of the . 
decision maker '.s own value system. Such judgments must come, at 
least indirectly, from man. 

The second observation is that to specify one's preferences 
objectively is not necessarily an easy thing for an individual to 
do. Even when all of the action alternatives have been made ex- 
plicit and the outcome of each possibility is known— that is, even 
when uncertainty is minimal — the decision task may still 'be a very 
difficult one. This is partiqularly true when the worths of tThe 
possible decision outcomes are intangible or depend on many favors. 
Consider, for example, the problem of choosing a house for purchase. 
-^Even assuming i:hat/one confines hi^ attention tc? a few houses that 
he knows are available, and that he has all the information that 
he wants about each one, he has the problem of somehow deriving 
from many factors (purchase price, number of rooms, design, ' general 
condition, extras— porch, garage, storage space, extra baths, 
fireplace— lot location and layout, distance from work, tax rate 
m town, services and public facilities in town) a common figure 
of merit m terms of which one house can be judged to be more or 
less preferred than another. 

In military situations, the specification of preferences may 
be especially difficult, it may often happen that none'of the 
possible decision outcomes is intrinsically desirable, and the 
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decision maker may find himself faced with the necessity of attempt- 
ing to choose the least undesirable one. The problem is aqqravated 
by the fact that the assignment of preferences may necessitate the 
weighting of time, materiel, territory and human lives. One balks 
. at _the^ Idea oX-tryj.ng.^tQ-speaif-y the value /of human lives and that 
of a piece of territory in terms of a common metric, but this is 
What IS done, at least implicitly, when a decision is made to at- 
tempt to gain a territorial objective when it is known that the 
endeavor IS likely to result in the loss of a certain number of men 
ihL private transportation system in the United States. 

The builders, users and regulators of . automobiles and highways have 
implicitly expressed a preference for a system that provides certain 
capabilities and conveniences at a cost of approximately 60,000 
traffic fatalities per^ar. One suspects that the exercise of 
making explicit how the various factors that contribute to human 
preferences are traded off against each other in specific decision 
sitdations would often be revealing to decision maker.- themselves, 

S?onSh ""^^ ^^""^ ^^"^^ conscious appreciation, v.lthout goiAg 

through such an exercise, of how such factors do combine to deter- 
mine their own preferences. 

^-h.-c ^°^5j:he e'ight aspects of decisiorr making in terms of which 
this report IS organized, preference specification is one of the 
two (the other is hypothesis evaluation) that have received the 

T^?^""^ °^ attention from. philosophers and researchers 
aiiKe. In the case of decision making under certainty, the studv 

tL^n l^""^^^ u?'^ ^^"'^^ °^ behavior amount to the same 

thing. Presumably one chooses what one prefers— and vice ver sa- 
lt he can know for certain what the decision outcome will be. 

9 .2 Some Early Prescriptions for Choice 

V 

r-^.r.J'i J^*^^"" ^° T^^^ Choices aitiQng alternatives that differ with > 
respect to several incommensurate variables, ,one must, at least 
implicitly, derive from the several variables involved a single 
figure of merit with respect to which the alternatives can be 
compared. That is to say, one must be able to decide that in some 
global sense Alternative A is preferred to Alternative B. How this 
IS -generally- done is not known; how it should be done is a matter 
of some dispute. Undoubtedly, individual methods for dealing with 
the problem range from highly intuitive impressionistic approaches 
(I Dust consider all the factors and decide that I like this com- 
bination better than that) to formal quantitative algorithms. 

. ^;^f^3amin Franklin was familiar with the problem, and his way 
of dealing with It IS at least of historical interest: "I cannot, 
fj^'voi^^i^LrT Premises, advise you what to determine, b'ut 

If you please i will tell you how. . . My wayn:?~to divide half a 
Sheet of paper by a line into two columns; writing over the one Pro, 
and over the other Con. Then, during three or four days' consideH^ 
tion, I put down under the different heads short hints of the 
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different motives, that at different times occur to me for or 
against the measure. When I have thus got them all together in 
one view, I endeavor to estimate the ^respective weights, [to] find, 
at length where the balance lies... And, though the weight of 
reasons cannot be taken with the precision of algebraic quantities, 
yet/ when eacly is thus considered, separately and comparatively, 
and the whole matter lies before me, I think I can judge better, 
and am less liable to make a rash step; and in faqt I have found 
great advantage for this kind of equation, in what may be called 
moral or prudential algebra ."* 

A more formal attempt to procedurize choice behavior was made 
at about the same time by the British philosopher and social re- 
former, Jeremy Bentham. Starting with the ba^ic premise that 
choices should be dictated by the extent to which their outcomes 
augment or diminish the happi'riess of the party or parties whose 
interest is in question (the "principle of utility"), Bentham 
attempted to define a quasi-quantitative procedure — a "hedonistic 
calculus" — the use of which would assure that the choices that are 
made would be consistent with this principle: 

"To take an exact account then of the general tendency 

of any act by which the interests of a community are affected 

proceed as follows. Begin with any one person of those whose 

interests seem most immediatjely to be affected by it,, and 

taJte an account: . , ' 

» * • 

(1) 0^ the value of each distinguishable pleasure 
which appears to be produced by it in the first instance. 

(2) Of the' value of each pain which appears to be pro- 
duced by it in the first instance. ^ 

I (S) Of the value of each pleasure which appears to be 

produced by it after the first. This constitutes the fecundity 
of the first pleasure and the impurity of. the first pain. 

(4) Of the value of each pain which appears to be pro- 
duced by it after the first. This constitutes the. fecundity 
of the first pain , and the impurity of the first pleasure. 

(5) Sum, up all the values of all the pleasures on the one 
side, and thoi^e of all the pains on, the other. The balance, 
if it be on the side of pleasure, will ^ive the good tendency 



*This account of Franklin's approp.ch to decision making was quoted ^ 
by Dawes and Corrigan (1974)/ who found it in -a letter from Franklin 
to his friend Joseph Priestly, dated September 19, 1772. 
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Of ,the act upon the whole, Vith respect to the interests of 
that i ndividual person; if on the side of pain, the bad 
^ tendency of it upon the whole. 

(6) Take an account of the number - of p_er_sons whose 
interests appear to be concerned, and.re^^at the above pro- 
cess with respect to ^each. Sum up the numbers expressive 
of the degrees of gogd tendency which the act has, with re- 
spect to each individual in regard to whom the tendency of ' 
It IS good upon ^he whole; do this again with respect to each 
individual m regard to whom the tendency of it is bad upon . 
the whole. ^ Take the balance ; which, if on the side~of r 
pleasure , will give the general good tendency of the. act, ■ - 
with respect to the total number or community of individuals 
concerned; if on the side .of pain, the general evil tend ency, 
with respect to the same community" (Bentham, 1939, p. 804). 

The value of a pleasure or pain, Bentham assumed, 'voUld depend 
on four factors: . 

" (1) Its intensity . r - ■ 

(2) Its duration. „ 

(3) Its certainty or uncertainty . 

(4) Its propinquity or remotenes s. " 

. Bentham did not expect that the procedure he defined would be 
strictly pursued previously to every moral judgment, or to every 
legislative or judicial operation"; but he did contend that it - 
represented a model of how judgments should be made, and a stan- 
dard against which wha'tever procedures are used might be evaluated. 

Bentham 's approach to choice behavior can be, and has been,' 
criticized on philosophical .grounds. The principle of "the greatest 
pleasure for the greatest number" is itself open to criticism, 
because it appears to place no limits on the extent to' which the 
many can prosper at the expense of the few, provided only that the 
'battom line" of the calculation of the net happiness is increased 
in the process. For our "purposes, the important point is_the fact„ 
— that Beirtham- a1:temptGd to' reduce the process of making Vhoices to 

a stepwise procedure, 

9.3 Simple Models of WoSrth Composition 

Although he used language that suggested that he believed 
that worth could be quantified and his procediire formalized as j, 
a §ort of calculus for computing the' worth of any given decision 
outcome, Bentham did not himsa^lf express his notions in mathe- 
matical form. His conceptualization of the choice process, how- 
^ . . ever, is clearly suggestive of a linear model which expresses the 
ipi^ \worth of a decision alternative as a function of the sum of the 
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values of the various components of pleasure (or pain) that that 
alternative represent^s, weighted by the number of .people that 
would be affected by the decision outcome. The choice would, of 
course, be iSie alternative with the greatest calculated worth. 
>« • 

Implicit in Benthaifi's presc^ription is the assumption that 
the total worth of a decision outcome is a monotonically increasing 
functipn of each of the factors which contribute, to the worth, and 
that the monotone character of this relationship for any given 
factor is independent of the values of the other factors. Yntema 
and Torgprsbn (1961) have suggested that there are probably many 
practical choice situations in which this is a'' valid assumption. 
For example, the wprth of a vocational choice probably increases 
monotonically with the attractiveness to the individual of the ■ 
work involved, whatever the status of the other factors to be con-' 
sidered. Yntema and Torgerson present some data that suggest 
that when this is^the case, the decision maker's choic*. behavior - 
can often be matched, if not improved upon, by a selection algo- 
rithm that takes account only of how worth relates to eaxjh of the 
factors individually and ignores the ways which the factors 
interact. To develop such an algorithm it is necessary only to 
determine how worth varies" with the individual factors. Several 
ways of making. this determination 'are 'suggested . An important 
point for Dur^purposes is that the relationships o'f interest may 
be inferred from the behavior of the decision maker when confronted 
with the task of choosing between pairs 6'f hypothetical alternatives 
selected to represent specific (in particular, extreme*) combina- ' 
tions of \the relevant factors.. 

Dawes and Corrigaii (1974) have recently taken an even stronge/' 
positibn with respQct to the^racticality and, the validity of simple 
linear decision algor.ithms in'a wide variety" of choice situations. 
They have shown that if, each of the factors contributing to the 
wprth of a decision outcome has a conditijbnally monotone* relation- 
ship, to that worth, and the measurerhent of these factors is subject 
to error, then not only are decisions that are based on weighted 
linear combinations of the factors likely to be better than those 
mada by human decision makers, but "in some cases this is true even 
- i-f -the weights are equal^'or randomly chosen . Data from several 
studies of judgmental and choice behavior are reviewed in support 
of this conclusion. Of the situations neviewed by Dawes and Cor- 
rigan, the only ones in which a linear weighting algorithm did 
more poorly than a hftiman decision maker were those in which the 
human's judgment- was based on information not taken into account 
by the algorithm. 



*A conditionally monotone relationship is one that is monotone, or 
can be made monotone by a scaling transformation. 
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9.4 The Problem of Identifying Worth Components 

The implication is that if one can identify the factors in 
terms of which worth is determined, one frequently can improve 
significantly upon liuman judgments by application of a simple linear 
' model. /The problem, according to this view, is not in the develop- 
ment of arcane mathematical decision algorithms,^ or even :ln th^ 
application of complex weighting functions to, a linear conUDination 
rule, but that of identifying the dimensions o^ the choice* space 
and of determining how these dimensions relate^ individually, to 
the wortji of the possible decigion outcomes. 

The danger in this line of reasoning is that of assuming that 
identification of the factors in terms of which judgments are, or 
should be, made is a trivial task. As we have already suggested, 
such an assumption is almost certainly false for many, if not most, 
real-life decision "situations. Most people can proba. ly recall 
choices that they have made which they realize in retrcpect were 
made without consideration of some factor that they would have 
recognized as relevant and important if only they had thought to 
think of it. An ^ individual buys a house, for example,^ and realizes 
too late that he failed to determine whether the cellar leaks. ^ 
Had the question occurred to ihim, he would have recognized it noHk 
only as a relevant consideration but as one that would have figured 
heavily in his assessment of the relative worths of candidate pur- 
chases. A potentially important aid to a decision maker would be 
a procedure that would facilitate the identification of the dimen- 
sions of his choice space. Having determined the factors upon 
which the relevant worths of possible choices depend, and how. these 
factors relate functionally to worth, a simple linear model of the 
type espoused by Yntema and Torgerson might then, be used to infer 
the decisionrvnaker 's behavior 'in a choice situation. The experi- 
mental results reviewed by Dawes and Corrigan suggest that such a 
model mif^gjit even be used in place of the decision maker to effect 
the choice. . s 

9.5 Studies of Choice Behavior 

In using the choices of a human as the standard against which 
to compare the performance of a model, one is assuming that humans 
behave in at least a consistent, if not an optimal, fashion. Only 
recently has the assutnption that decision makers are able to make 
consistent choices amopg alternatives that differ on many di^men- 
sions without recourse to formal analytical procedures been tested. 

Sflovic and Lichtenstein (1971) have re\Aiewed several' approache 
jthat have been taken to the problem of describing how people do in 
fact make such choices. They divide these approaches into two. 
major categories: those that make use of correlational or regres- 
sion Analysis or the closely related analysis of variance, ^^d thosi 
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that .'Hike use of Bayes theorem. Among the nonBayesian approaches--, 
that are reviewed are the correlation model of Hoffman (i960; 1968V, 
the lens model of Brunswik (1952, 1956), the integration theory of 
Anderson (1968, 19-69), and the theory of conjoint measurement of 
Luce and Tukey (1964) and Krantz and Tversky (1971). The objective 
m all of this work is to d-is.cov^r and describe how a human "judge" 
combines information concerning different attributes of a choice 
alternative to. arrive at a judgment of its overall desirability 
relative to the other alternatives ^aong which a choice is to be 
made. 

The, results of many of the studies reviewed by Slovic and 
Lichtenstem (1971) suggest that, although people can make "wholistic 
evaluations" (Fischer, 1972), they tend to focus their considera- 
tions on less. than the full set of dimensions, and, as a conse- 
quence, frequently ignore potentially important information. Also, 
there appears to be a degree of random error in the evaluation 
process which increases as the decision maker attempts to consider 
increasing numbers of relevant attributes (Hayes, 19 64; Kanarick, 
Huntington, & Petersen, 1969; Rigney & Debow, 1966) . • 

. On the l^asis of results obtained in his study of job-seeking 
behavior, Soelberg (1967) challenged the idea that people generally 
do make choices m accordance with worth-calculation models in 
real-world situations. In his words, "The decision maker believes 
a priori that he will make his decision by weighting all relevant 
factors with respect to each alternative, and then 'add up num- " 
bers' m order to identify the best one. In fact, he does not 
generally do this, and if he does, -it is done after he has made 
an implicit' selection' among alternatives" {p~. W^t. Soelberg 
draws a number of other conclusions from his study which,' in the 
aggregate, seem to suggest that much of the effort that goes into 
decision making is calculated to rationalize—rather -than. arrive. 
at~a choice. It's as though the decision maker were in cahoots 
with himself to deceive himself into perceiving his choices as 
vje'll- founded when in fact the real basis for them may be unknown. 

9 . 6 Procedures for Specifying Worth 

Obviously people can—people do— make choices ' among multi- 
dimensional stimuli; the results mentioned above suggest, however, 
that our ability to handle many dimensions simultaneously in a 
consistent and reliable way without the aid of a formal procedure 
IS somewhat limited. _ Given that the problem seems to be one of 
exceeding man's ability to process information, it is not sur- 
prising that some of the solutions that have been proposed take 
the form of ways of restructuring unm'anageable problems so as "to 
make them into problems of simpler proportions. Such procedures 
are sometimes referred to as decomposition procedures because 
they divide the task into subtasks that presumably are within the 
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decisio/ mak^s information-processing capabilities. The solution 
to the^ubtasksXare then used as a basis for inducing a solution to 
the original prdblem. -luuxwu to 

These pr4:^res typically involve a number of steps (e g 
Fischer, 1972) ^uch as specifying the alternatives to be compared, 
specifying the dimensions or factors with respect to which the 
alternatives are to be compared, assessing the worth of each alter- 
native with respect to each dimension, and combining the results of 
the dimension-by-dimension assessments into SQjne overall indicant 
of worth for each alternative. The first of these steps has not ' 
been a focus of attention in studies of preference specification; 
the alternatives usually are provided. In the real world, identi- 
fying these alternatives can be a nontrivial problem, but it is 
perhaps better thought of as a problem of information gathering 
than one of specifying preferences. The second step Uso has not 
received much research attention. • f j. « adb ^ot 

\ 

A great deal of attention has been given to the third- of the 
steps mentioned by Fischer (e.g., Becker McClintock, 1967? coombs, 
Lhnlv .""^/""J ^^^2' Fishburn, l'967; Hammond, 1967; 

Sahney, & Ford, 1969; Luce & Tukey, 1964; /MacCrimmon , 1968; MiUer^ 
.Kaplan, s Edwards, 1967; Raif fa, 1968) . Numerous techniques have 
been proposed and studied for assessing the worths of alternatives 
Zt^^ respect to individual dimensions or factors. These techniques 
range from simple, qualitative pair-comparison procedures that 
yield ordinally scaled preferences to relatively complex methods 
for deriving ratio scales for interdependent factors. 

MacCrimmon (1968) has reviewed several prescriptive techniques 
for choosing among alternatives that differ with -respect to multiple 
factors. The efechniques that he considers are discussed under the 
^^if^r^'^.J? Joininance, (2) satisficing, (3) maximin, (4) 

indiv '^RpL-f i''^?^^^^^' ^ddiUive weighting, (7) effectiveness 

index, (8) utility theory, (-9) tradeoffs,, and (10) nonmetric Scaling 
iLn?™''^^' h^^'^^s^^^bfs the necessary assumptions and information 
requirements, and presents a forma], mathematipal representation of 
the optimal (or best) choice defined by the technique. Considera- 
tion is also given to the possibility .of using several methSds, in 
Pil?kera967r ^ ^""^^^ choice problem, as suggested earlier by 

A more recent review of worth-assessment techniques has been 
prepared by Kneppreth, .Gustaf son, Leifer, and Johnson (19740 > In 
this review, methods are classified in terms of five properties:. 
(1) whetner probabilities are used, (2) what kind of judgment is 
ulTi,T^ (e.g., simple preference, numerical assignment), (3) num- 
fn^ ^Lt^^ involved m a single judgment, (4) whether appropriate 

for continuous or discrete factors,, and (5) nature of output pro- 
duced (e.g., j-ankmg of worth, quantitative indicant, o! worth) 
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, Especially helpful features of this review are explicit discussions 
of what the authors see^ as the primary advantages and disadvantages 
associated with each of the methods described, and the provision of 

^ references for the theoretical bases of these techniques. Of par- 
ticular relevance to this report is the stress that Kneppreth, et al. 
put on the need for training before some o£ these techniques cau be 
used effectively. 

The -fourth 3tep irjenticned by Fischer — that of combining the 
results of factor-by- factor ass^essments into overall worth esti- 
mates — has proven not be a difficult one in many practical situa- 
tions because of the fact that a simple 'linear combination rule 
seems to work remarkably well in so many cases (see Section 9.3). 

Prescriptive techniques for preference specification, or worth 
assessment, are of considerable interest because of the potential 
that they represent for procedurizing — and thereby, hopefully, 
simplifying — the solutions for complex choice problems, A less 
tangible but perhaps no less important benefit that can result from 
attempts to apply such prescriptive techniques in real-world situa- 
tions sterfis from the fact that these procedures , force the decision 
maker to be explicit concerning his own value system as it relates 
to the problem at hand. This fact has obvious ramifications vis-a- 
vis the problem of evaluating the performance of decision makers 
who make choices that affect the lives of others; one clearly wants 
tcknow, in such cases, not only what the choices are/ but the 
bases on which they are made. .Being forced to be explicit concern- 
ing the factors that determine his choice ind the relative importance 
tfiat he attaches to each of"^hem may be as revealing to the de- 
cision maker himself as to an independent observer. 

9.7 Preferences among Gambles ^ \ 

So far, we have considered only the problem of specifying 
preferences among stimuli that differ perhaps in many, but in known, 
ways. In this cas,e the decision maker knows what the effect of any 
choice that he may make will be. Another type of preference speci- 
fication that has been studied involves preferences among gambles, 
or between gambles and " sure things ^ " The general procedure in 
such studies is to present the decision maker with a choice, either 
between two wagers, or, more typically, between a wager and a sure 
thing, and then to adjust either the possible outcomes of the 
wager (s) or the probabilities of these outcomes until the decision 
maker is indifferent to the alternatives from which he must choose. 
By repeating this process a number of times with different v;agers, 
onfe can generate the kind of data from which worth functions can 
be inferred. 
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Topically, -the wagers that have been used in these studies 
are such that one of th^ possible outcomes^is more desirable than 
the other, and the probability of the less'^desirable outcome is 
unity minus the probability of the more desirable one. siovic, 
(1967, 1969), however, has studied preference behavior in so-called 
duplex gambles in which the probabilities of "winning" or "16sing" 
can be varied independently of -respective payoffs. In, this situa- 
tion, the decision ijaker can win and not lose, lose and not win, * 
win and lose, or neither win nor lose. As Slovic points out, "It 
can be argued that this type of gamble is as faithful an abstrac- 
tion of jeal-life decision situations as its more commonly studied 
counterpart m which the probability of losing is equal to unity 
minus the probability of winning (p, = l-p ) . For example, the 
choice of a particular job might offer somg probability (p^) of a 
promotion and some probability (p ) of a transfer to an uncfesirable 
location, and it is possible that one of these event", both o-p < 
them, or ^neither of them, will occur" (p. 223) . 

. I", ■the first of Slovic 's studies, two different methods of 
indicating the attractiveness/unattractiveness of a' wager were^ 
explored. One method required the subjects to rate strength of 
preference directly 'on a scale rangii^g from +5 (strong preference 
for playing; to -5 (strong preference for not playing). The second 
required the subject to equate the attractiveness qf this gamble 
with an amount of monkey such that he would be indifferent to play- 
ing the gamble or receiving the stated amount. One third of the 
subjects assigned to the second method v/ere required to state the 
largest amount they would be willing to pay the experimenter in 
order to play each bet, and, for an undesirable bet, the smallest 
amount the experimenter would' have to pay them -before they would 
p^'ay it. Another third of the subjects were given ownership of a 
ticket for each gamble and required to state' the least amount of 
money for_which they would sell the ticket. The subjects in the 
final third were required to state a fair price for a given gamble 
m the absence of information as to whether they or the experimenter 
owned the right to play it. 

Slovic demonstrated tfiat subjects did not weight the risk 
dimensions in the same way when bidding as when rating. Variation 
in the ratings was influenced primarily by variation in- probability 
of winning (p^)5- while variation in bidding was influenced primarily 
•by variation In probability of losing (p ) . Also, payoff dimen- 
sions—dollars won ($W) and dollars lost^($L) produced more effect 
on bids than on ratings, while probability dimensions produced 
more effect on ratings than on bids. Finally, it was found that 
when a person in the bidding group considered a bet to be attractive, 
his judgment of its degree of attractiveness was determined pri- 
marily by the amount ($w) ; when he disliked a bet, the primkrv 
determinant of the degree of dislike was ($L) . This finding has 
particularly important methodological implications, because, as 
Slovic points out, no existing prescriptive theory of decision 
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making wou^d 'consider that response mode should be a" determinant of 
?n in which decision makers utilize probabilities and payoffs 

decisions. under risk, and he argues that behavior in sGch 
co™r':?i:ns"'" influenced ly information!p?ocessSf 

Preference Specification and Training 

• *. °^ ^^^^^ thought, preference specif ication—amona all fhP 
leas? cha'nenaf making-might^ appeal?? pose'^he 
ittr^ f thito^^ training research. One might, assume 

that If there xs an aspect of decision making that comes 

?hSgs cLaj^v'a?."' r^^^^ ^^^^ preferences are. 

abundJnt ^h^i^ are not that , simple,, however, ' and the evidence is 

Sr af ?Ls^ ^or?o^nf?^?^.^'^^^" ^^^^ their -preferences are, 

or at least how to speciIT-them ■ in an unambiguous and .::onsistent 

" problems JhirjSLjr^^'^f -^^^^^ f^^'^^^ suggests at least four 
Fi?st is ^ho ^- t° t^^^^i^g and preference specification. 

sibieciive nrS^JSt?? ^° ^^^^^ "^^J^^ judgments of 

Sk?! 5 ^ probability that are independent of the worths of pos- 
sible decision outcomes, as the use of subjective e^pectld utility 

qSestior?r?ha\^orH''"r'°f ^ ^^^^ ctTeTLTatla^^ 

question is that of how to train people to make worth judgments 
that are invariant across different measuring techniques? 

bv aJ^ni■tZll°^'^^''^ of decomposition methods has been motivated 

^ ^ si"»Plifying the process of making preferences 

Johnsorn97lf'H Kneppreth, Gustafson,'^£elle?? Ind 

Dar?K,niii3 however, some of these procedures, 

x^J J^T Y that^eld the most quantitative results are ■ 

workable only with relatively sophisticated users! A third 'challenge 
Sdi^a^'ir^"^ research, therefore, is to develop methods for p^o- ^ 
viding the necessary training in cost-effective ways. 

A fourth problem relates to two aspects of decision makir^rt 

s^ulfefof information gatheri'g?'%'°"i:ary 

stuaies Of choice, the dimensions in terms of which preferences 
are to be" specified typically are given. In real-worW situations 
however, the dimensions of choice are often determined bvthS 

sideJeTi^at? '-'r""; factors'that'^a^fcon- 
Ii?e?nftJvefa?e^tS?;-''?H^r!r ^f^^tive merits of the cho?Se 
u I ; those that the decision maker happens fr> ^-hinV 

to°t^; ^^^P^^i^i^fly little attention has been given by ?esea?chers 
the ?ac?ors iha.°fH^°^ enumerating on demanJ 

choicfSuaSon rl 7°^^'^,^°^^^^^^ important in any particular 
a list of suci ^""^^ whether, when provided with 

lis^ is comn?^^f S """^ ^^^^ confidence whether the 

list IS complete. Much more research is needed, both to determine 
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human limitations and performance characteristics in this regard, 
and to explore how training might improve one'^ ability .to make 
one*s worth space explicit vis-a-vis specif ic , choice problems. • 
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' SECTION .X 

ACTION SELECTION 

Selection, or choica, is often thought of as representing the 
essence of decision making. And obviously, if one has no options, 
then he has no decisions to make. Paradoxically, however, the act 
- of choosing per se "is the least interesting of the aspects of 
decision making that are considered in this report. This is be- 
'cause of the fact that when the other aspects h^ve been realized— 
when information has been obtained, the decision space structured, 
hypotheses generated and evaluated, and preferences stated— the 
choice may, m effect, have been *;aetermined. This is, of course, 
as It should be. One's goal in all of these activities is to remove, 
insofar as possible, doubt ^about what the choice should be. 

In spite of his best efforts to reduce, uncertainty to a minimum, 
and thereby to discover what his decision ought to bef however? the 
decision maker may, on occasion, feel. very much "l4ft to his own 
devices when forced to make a choice,. Ellsb6rg (1961) rather 
graphically described the sense of frustration that one can feel 
when he faces his moment of truth and is not entirely convinced of ' 

"^^H^f^-^^^ ° o.^^! ^f^^^ ^^^^^ choice will have to be made. 

(This) judgment of the ambiguity of one's information of the over- 
all qredibility of one's composite estimates, of one*s confidence 
m them, cannot be expressed in terms bf relative likelihoods or 
events (if It could, it would simply affect the final, compound 
probabilities . Any scrap of evidence bearihg on relative 3,ikeli- * 
hood s-hould already be represented in those estimates. But having 
exploited knowledge, guess, rumor, assumption, advi.:e, to arrive 
at a final judgment that one event is more likely than another or 
that they are equally likely, one can still stand back from this 
process and ask: 'How much, in the end, is all this worth? How 
much do I really know about the problem? How firm a basis for 
choice, for appropriate decision and action, do.J have?.' The 
answer, 'I don't know very much, and I can't rely on that,' may 
sound rather familiar, even in connection with markedly unequal 
estimates of relative likelihood, if 'complete? ignorance ' is rare 
or non-existent, 'considerable' ignorance is surely not" (pp. 20,21)^ 

Most of the decision situations that we have considered in 
this report involve the problem of choosing one from among several 

t 

~~' ■ I ■ ■ III ^ « 

*This statement is contained within a larger discussion of circum- 
stances m which it may be "sensible" to act in conflict with the 
prescription of the Savage (1954) axioms (see Section 2.2) The 

^"^^ discussion for an interesting anal- 
ysis of the problem of ambiguity in choice behavior. 
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courses of action. It is important to note, however, that people 
sometimes find themselves faced with the task of deciding not what 
to do, but when to do it^ ^ The required action may be dictated 
by circumstances, or predetermined in one way or\ another, but 
the individual is left with the job of deciding on the best time 
to act. This -type of decision problem is nicely illustrated by 
the following, situation. ^ * • 

Consider a pistol duel in which the duelists are instructed 
to turn to face each other on signal and to fire one shot at will. 
Suppose that once the men have faced each other, each may walk 
toward the other, reducing the ^distance between them if he wishes. 
We may assume .that the accuracy of each duelist improves, although 
not necessarily at the same rate, as the distance between them' 
decreases i Clearly, each man fac.es a dilemma: every second that 
ha delays firing in order to decrease the distance between him and 
his opponent and to increase his chances of an accur^ te ^hot, he 
also increases the chances of success for his opponent; on the 
other hand, if he fires too soon; he risks missing, in which case 
his opponent is free to advance on him until his shot will be cer- 
tain to find" its mark. 

This type of situation is representative of what Sidorsky, 
Hoiifeeman, and Ferguson C1964) have characterized as "implementation 
type decision tasks." In. ^J-^^o^sky ' s experiments the duelists v;ere 
simulated navy tactical units, htt the problem was essentially the 
same as that of the individual antagonists. The decision maker 
had to decide when to fire a missile, knowing that bo^h the proba- 
bility of hitting his opppnent and the probability of being hit by 
him were increa'sing (but at different rates) in time. 

A particularly interesting result from this work is the find^ 
ing that subjects performed less appropriately when operating at 
a disadvantage than when operating at an advantage. One of the 
conclusions that Sidorsky and his colleagues drew from the results 
of a series ^of studies (Sidorsky & Houseman, 1966; Sidorsky, Horse- 
man, & Ferguson, 1964; Sidorsky & Simoneau, 1970) was that "the'' 
inability to analyze and respond appropriately in disadvantageous 
situations isf a major cause of poor performance in tactical de- 
cision making" (Sidorsky & 'simoneau, 1970, p. 57). If this obser- 
vation is _ generally valid, i^'s implications for tactical decision 
making are clearly very significant. The implications for training 
are also apparent, namely, the need for extensive decision-making 
experience in disadvantageous situations. 
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SECTION XI . . . 

DECISION EVALUATION 

The problem of evaluating the performance of decision makers 
is a difficult one and it is critically important to the task of 
training. Without an evaluation scheme, there is no way of as- 
certaining whether training has resulted in an improvement in 
decision-making performance, Trainii^g assessment is not the only 
reason for an interest in evaluation of decisions-making performance, 
however. Anyone who finds himself in d position of having to pass 
judgment oa the performance of a decision maker is in ne'ed of a 
set of criteria in terms of which that judgment can be made. More- 
over, a decision maker himself might wish to evaluate a particular ' 
decision that he has made in terms of a set of objective criteria. 

Unfortunately, a completely satisfactory set of^^jective 
criteria against which performance can be compared has not been 
developed. As Kanarick (1969) has pointed out, ^'unlike other 
behaviors, there is no standard dependent variable, such as time- 
on-target, trials to criterion,, or percent correct. " One can, of 
. course, choose for study in the laboratory only tasks for which 
performance can be objectively evaluatecJ (e.g., probability esti- 
mation for frequentistic .events); however, one runs the risk of 
thereby excluding from study a large percentage of the problems of 
interest. Certainly, in most real-life decision situations in 
which the objectives are complex,, the stakes are real, and the 
information is incomplete, evaluation is an extremely difficult 
ta^. ^ . 

11*1 Effectiveness versus Logical Soundness 

Of central importance to a discussion of evaluation of de- 
cision making is the distinction between effectiveness and logical 
soundness. Failure to make this distinction sharply — sometimes 
to make it at all — has resulted in much cbnfusion in. the litera- 
ture. Effectiveness and logical soundness are quite different 
things. "One might be willing to assume that logically sound de- 
cisions will, on the average, tenS to be more effective than 
decisions that are not logically sound. However, the assumption 
that the correspondence will necessarily hold in any. particular 
instance" is manifestly not valid. 

A decision is effective to the extent that the result to 
which it leads is one which the decision maker desires. Effective- 
ness usually is easily determined after the fact. The logical 
soundness of. a decision depends on the extent to which the de- 
cision maker's choice* of action is consistent with the information 
available to him at the time the decision was made, and with the 
decision maker's own preferences and goals. That these are quite 
different factors is clear from a simple example. Suppose that 
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one is given the option of betting $5 against $20 that the next 
roll of a fair die will come up 6, or betting $10 against $12 that 
the up face on the next roll will have an odd number of dats. If 
he elects to make the first bet and the roll produces a 6, we would 
say^ that the decision was an effective one. However^ whether it 
could be considered a logically sound one would depend on what the 
decision maker's objectives were. If his intent vas to maximize 
his potential gain^ or to minimize his potential loss, the decision 
was sound. If his intent was to maximize his expected gain, it 
was not. 

Decision-making behavi^o:^. ^should be evaluated in terms of its 
logical def ensibility and not in terms of its effectiveness, inas- 
much as effectiveness is found to be determined in part by factors 
beyond a decision maker's control, and usually beyond his knowledge 
as well.* ^ It often. appears not to work this way in practice, how- 
ever. Evaluation of decisions in terms of their outCv.mes seems 
to be the rule, for example, in the world of finance and business* 
Investment counselors are hired and fired on the basis of the con- 
sequences of their portfolio recommendations, ^and corporate manage- 
ments are frequently juggled as a result of unsatisfactory profit 
and loss statements, Although the cliche "it's the results that 
count" has particularly strong intuitive appeal in this context, 
decision outcome is no more justified as the basis for evaluation 
of decision making in the financial world than in any other. As 
Krolak (1971) asserts in a discussion of portfolio management 
evaluation: "The real question to be answered is how well did [I] 
do with the information, capital, strategy and ability to assume 
risk as compared with others who might possess the same resources?" 
{p. 235) . 

That decision-making performance in military-training situations 
is not always evaluated in terms of its logicality, has been noted 
by Hammell and Mara (1970). In discussing some of the mission 



Commenting on Fuchida and Okumiya's account of the WWII Battle of 
Midway, Admiral Spruance (1955) made the following interesting 
observation: "In reading the account of what happened on 4 June, 
I am more than ever impressed with the part that good or bad for- 
' tune sometimes plays in tactical engagements. The authors give us 
ciredit^ where no credit is due, for being able to choose the exact 
time for our attack on the Japanese carriers when they were at a 
great disadvantage— flight decks full of aircraft fueled, armed 
and ready to go. All that I can c^laim credit for, myself, is a 
very keen sense of the urgent need for surprise and a strong 
desire to hit the enemy carriers with our full strength as early 
as we" could reach them." 
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training that* is carried out in ASW tactical training programs, 
they point out that performance evaluation is based, in many in- 
stances, on the simple ef f ec^^ivfetiess indicator of whether or not 
the team scores a hit* If it does, performance is judged to be 
acceptable. Commenting on specific training exercises that they 
observed they note: "If a hit was made, regardless of circumstances, 
each team member's performance was usually .considered good... In 
some instances a hit was scored because the target would make a 
predet'ermined maneuver into" the path of a torpedo which had been 
obviously fired in a wrong direction" - (p. 9). 

' It is probably safe to assume that most people in decision- 
making positions are more likely to be rewarded, or censured, as ^ 
the case may be, on the b^is of the effectiveness of their de- 
y cisions than on that of their logical quality. This is due in 
part perhaps to the fact that society is far more interested in 
• the results produced by its decision makers than in the reasons 

for which depisions were made, it is undoubtedly ^Isb true, however, 
that it is easier to determine the outcome of a decision than to ^ 
determine ^ whether the decision was logically justified at the time 
that it* was taken* One wonders how many heroes have been made, not 
in spite of, but because of, very po&r decisions which have had 
happy outcomes, and, conversely, how many "bumblers" owe their 
reputations not to the illogicality of critical decisions they 
have made, but to fortuitous turns of events that have blessed 
sound choices with disastrous results. 

We may note in passing that even if one wishes to evaluate a 
decision in terms of its effectiveness, rather than its logical 
soundness, the task may be less than straightforward. Miller and 
Starr (19 69) make the point that decision objectives are not always 
singular. Often, one is attempting to realize several objectives 
simultaneously, and seldom is it possible to optimize with respect 
to all objectives at the same .time, it is difficult in such cases 
to evaluate a decision outcome unless' its implications with respect 
to all the objectives can be combined |nto a single figure of merit. 

One attempt to develop a procedure for combining performance 
scores on various decision-effectiveness criteria into a single 
figure of merit was made by Sidorsky (1972), and Sidorsky and his 
colleagues (1968, 1970). A set of operational criteria that were 
intended to be used to evaluate the decision performance of a 

? military tactical unit was identified as follows: spatial rela- 
tionships (the spatial interface between own and enemy tactical 
units/, self-concealment (the degree of success in keeping the 
enei^ uninformed concerning ^own unit) , information generation 
(the)/ decree of success in keeping informed concerning enemy 
unit!*), weapon utilization (desti.oy or counterattack capability), 
ard conservation of resources (adequacy of supplied) . Such 
criteria have been used by Sidorsky to rate the quality of 
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decisions ina*de during experimental tactical exercises. A 
Decision Response Evaluation Matrix was developed which, when 
used in conjunction with an algorithm for combining scores with 
respect to all five operational criteria, permitted jthe quality 
of a decision to be expressed as a single measure. 

11.2 Evaluation Criteria 

Granted that logical soundness is the appropriate basis on 
which to evaluate decisions, the problem then is to translate 
that principle into a set of Objective criteria against which ' 
decision-making performance can\be judged. In view of the huge 
literature on decision making^ surprisingly little attention 
has been given to thip problem. 

Sidorsky and his. colleagues (1964, 1966; 1968, 1970) and 
Hammel Land Mara (1970) have suggested five behavioral factors in 
terms of which an individual decision-maker's performance might 
be judged: stereotopy (the tendency of a decision maker to respond 
in an unnecessatily predictable way) , perservation (the tendency 
to persist when persistence is unwarranted)^, timeliness (the 
^extent to which the decision-maker's behavior is reasonable in 
terms of the time constraints imposed by the situation) , 
completeness (the extent to which all available relevant informa- 
tion is used), and series consistency (the consistency of the \ 
decision-maker's behavior within the context of a series of 
interrelated actions) . The first two factors are liabilities 
for a decision maker; the last three are assets. In contrast 
with the operational criteria mentioned in the preceding section, 
these behavioral criteria are more concerned with the logicality 
of • a decision than with its effectiveness. 

The conceptualization of the decision-making process that has 
provided the structure of this report suggests a number of dimensions 
with respect to which the quality of a decision-making activity 
might be evaluated: ^he adequacy of the inf?:)rmation-gathering 
proces^; the sensitivity of data evaluation; the appropriateness 
of the structure that is given to a decision problem; the facility 
with which plausible hypotheses are generated; the optimality ot 
hypothesis evaluation; the sufficiency with which preferences 4-^^/ 
specified; the completeness of the set of decision alternative^^ / 
that is considered; the timeliness" of action selection and ite 
consistency with the decision maker's preferences, objectives /and 
information in hand. The development of techniques for assessing 'X 
tha^e aspe-rts of decision making quantitatively and unambiguously 
represents a challenge to investigators of decision-making behavior. 

11.3 A Methodological Problem 

It is worth noting that to determine after a decision ijas been 
made whether its basis was logically sound may be a very difficult 
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task. People usually can give plausible reasons for choices thev 
have made. One may be permitted a certain amount of skepticism, 
however, concerning whether reasons that are given after the fact" 
are _ the reasons that prevailed at the time of the makinq of the ^ 
choice (Soelberg (19 67), This is not to suggest that people neces- 
- sarily misrepresent the bases for their decisions intentionally. 
It seems nob unlikely, however, that we frequently convince our- 
selves, without being conscious of doing so, that choices have 
been determined by certain rational considerations, when in fact 
those considerations were discovered or invented only after the' 
choice was made. One might argue that even though the alleged 
basis of a decision may not have been verbalized, or even consciously 
appreciated by the decision maker, it could still have been opera- 
tive at a subconscious level at decision time. But this is a 
dirficult, if not impossible, point to confirm or invalidate ex- 
p!=i=rn^Qi^^( ^^^^ r^^^oti it is not a very useful hypothesis. 

Fascai (1910) expressed his skepticism concerning the credibility 
ot after-the-fact introspective explanations of behavior over three 
Il^^^^c Jr""! ago: "M, de Roannex said: 'Reasons come to me after- 
wards but at first a thing pleases or shocks me without my know- 
ing the reason- and yet it shocks me for the, reason which I only 
discover afterwards.' But I tjelieve, not that it shocked him for 
the reasons which were found afterwards, but that these reasons ' 
were only found because it shocks him" (p. 98). 
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, ' SECTION- XI-I 



SOME FlfR*HER COMMENTS ON TRAINING OF. DECISrON MAKERS 

• ' • • '"\ ' ■ • " ■ * 

Throughout this report we have commented: on how the theoret- 
ical notions and research findings that have ieen reviewed relate 
to issues of training and training research. . f hese comments have 
been made within the contexts .pf the discussions -to which they A 
pertain. It is not our purpose in this sectioh to review or sum- 
marize these comments, but rather to turn to sime training-related 
topics that hav.e not been addressed easewhere L the. report. ^ 

^2-1 Performan ce Deficiencies versus PejrfCriimnci Limitations ' 

'■ ""-"-^ ~' ' ^ i" 

Some investigators (Hammell & Mara, 1970) have advocated X 1 
the approach of identifying "behavioral deficiencies" and- ^ \ 
developing training programs that are designed to ameliorate ' 
«f^a>rftntiS''^^' Kanarick (1969) has suggested that bnilcomponehtl 
orf a training program for decision maker^ should be Chat of makinS 
themraware of some of the common reasons for the making of poor ' 
decisions. ' . i'^^-^ 

/The term "deficiencies" has been used in .two ways in the 
litetature: to refer to stereotyped ways of behaving suboptimally, 
and to refer to basic human limitations. In what follows, we will 
refer to the second type of "deficiencies" as limitatioiis-, and 
use the word -deficiency only to deno"Bfe suboptimal but presumably 
■correctable behaviors. An example of a behavioral deficiency 
would be the tendency of humans to be overly conservative in their 
application of probabilistic information to the evalualion of hy- 
potheses. A possible example of a limitation would bejthe in- 
ability of most people to weigh more than some small number of 
factors-, without some procedural help, iji arriving at a preference 
among choice alternatives. 

The distinction l^^tween deficiencies and limitations has 
important implications for training. Dsficiencies may be "trained 
out ; basic limitations must be "trained around." 

nr : problem in dealing Wth either a putative deficiency 

or a limitation, however, xs to verify that it indeed exists It 
iffJorS""^^ imperative, when a deficiency or limitation is iden- 

yi I single experimental study, that the finding be cor- ^ 
roborated by further research. More iif^prtant, howeve?, and more- 
^^.ff^c^ of establishing that 'the conciusJons 

drawn from experimental studies are valid beyond the laboratory 
environments in which the results were obtained. ?t is exceed-" • 
ingly difficult to capture some of the aspect^ of many rJal-world 

t?^;f°\T^'^r ^^^^ ^^^^ ^t^^^^) laboratory Situa- 

tions And what may constitute appropriate behavior in the one- 
situation may prove to be inappropriate in the other. 

<^ ' ' 
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Assuming, however, that one^ls able, to identify some examples 
of deficient behavior that appear to be fairly universal among 
decision makers, the question is how to go about training them out. 
One obvious possibility is to expose trainees to decision-making 
situations in which a given deficiency is likely to show itself 
if it is ever going to do so, and then provide the individual with 
some immediate feedback concerning the appropriateness of his be- 
havior ♦ One would probably want to provide numerous opportunities 
for the same deficiency to show itself in a variety of contexts, 
providing feedback to the trainee each time that the deficiency is 
displayed. Probably, too, feedback should be provided for some 
time after performance has improved to the point that the deficiency 
is no longer apparent. 

When dealing with basic human limitations, the goal should be 
to educate the decision maker concerning what those limitations 
are and to provide him with the means for working around them. 
For example, if it is the case that without the help of some ex- \ 
plicit procedure, a decision maker cannot effectively weigh more 
than n variables in attempting to optimize his choice "of an action 
alternative, it may be futile to try to train him to, make effective 
use of more variables; however, if that is the case, he should be 
made aware of his limitation and be trained to perform within it. 

> 

Another approach to dealing with deficiencies . and limitations — 
in addition to training — is that of providing the decision maker 
with aids' to facilitate various jaspects of the decision process. 
The goals of training and of dedision aiding are not viewed by the 
writers as mutually exclusive, but rather as complementary, ap-' 
preaches to the improvement of decision making. Moreover, the 
fact that decision aids are being developed has implications ,for 
training, a point to which we will return in Section XIII. 

12.2 Sim.ulation as an Approach to Training 

A common approach to the problem of training decision makers 
is that of simulation (Bellman, Clark, Malcolm, Craft, & 
Ricciardi, 1957; Cohen & Rehman, 1961). The idea is to place 
the decision maker in contrived situations that are similar 
in certain critical respects with the decision-making situations 
that they are likely to encounter in the real 'world. The approach 
has been used in efforts to train business executives (Martin, 
1959)7 prospective. high -school principals (Alexander, 1967), 
:<esearch and development project managers (Dillman & Cook, 1969), 

. military strategists and ^tacticians (Carr, Pyrwes, Bursky, Linzen, 
& Hull, 1970; Paxson, 1963) , high-school history and science 

^ teachers (Abt, 1970), vocational-education leaders (Rice & Meckley, 
1970), and government planners (Abt, 1970) ♦ 
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Most business colleges and graduate schools today make some 
use of simulation and gaming' tefchhiques to teach management and 
decision-making skills. Also, as a result of early "efforts by the 
American Management Association tq develop a decision-makinq 
course, corporations such as General Electric, Pillsbury, Westlnq- 
house, and Standard Oil. of New Jersey have (Revised in-house trainina 
programs that make use of simulation techniques. 

Two different forms 'of management-training games are discussed 
by Cohen and Rheninan (1961) in their survey of the present and fu- 
ture roles of such games in education and research. The first 

' fonii--the "general-management" game— attempts to provide experience 
in the making of business decisions at a top-executive level, while 
the second form--the "functional" business ^ame— focuses on specific 
decision situatlLons within a limited -functional level of the orqan- 
azatxon. Because of., the complexity of interactions among organi- 
zational entities and the multidimensional ity of the a^cision envi- 
ronment simulated in the general-management games, the possibility 
defining and utilizing optimal strategies has not yet been 

demonstrated. The functional game situations, .on the other hand, 
which are typically lower in complexity, allow for the specification 
and application of optimal or "best" strategies. 

A variety of views have been expressed concerning the strengths 
and weaknesses of simulation as an approach to training. Kibbee 
(1959) suggests the following advantages: - 

"1) It (simulation) can provide a dynamic opportunity for 
learning such management skills as organization, planning, 
control, appraisal, and communication. 

2) Simulation can provide an executive with an appreciation 
of overall company operations and the interaction between man, 
money and materials. It helps make a generalist out of a 
specialist who has never had the opportunity of reviewing his 
decisions as they affect the organization as a whole. 

3) Simulation can provide executives with practice, insight 
and improvement of their main function: making decisions. 
Faced with realistic decisions about typical business problems, 
they can experience years of business activity in a matter of 
hours, in an environment similar to that they face in evervdav 
life. 

4) Simulation can exhibit what Dr. Forrester of M.I.T. '-^Us 
the 'dynamic, ever-changing forces which shape the destiny of 

. a company. ' The general business principles that are illus- 
trated can be studied and understood by the participants" 
(p • 8) • 
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Similar theme^s are" expressed by Abt (1970) concerning the efficacy 
of management games: 

I. ' 
Games are effective teaching and training devices for stu- 

. dents of all ages and i-n many situations because they are / 
highly motivating, and because they communicate very effi- 
ciently the concepts and facts of many subjects. They create 
dramatic representatives of the real problem being studied. 
The players assume" realistic roles, face problems, formulate 
strategies, make decisions, and get fast feedback on the 
consequences of their action. 

In short, serious games offer ajs a rich field for a risk-free, 
active exploration^of serious intellectual and social problems" 
(p. 13) . ' 

Simulation, as a general approach to* training of decision 
makers is not without its critics, however. Martin (1959), who 
generally endorses the approach, volunteers several caveats. He » 
points out, for exampl^, that many of the qualitative dimensions * 
of a situation, such as "personnel quality and morale in an organi- 
zation being modelled, are difficult to reflect in a game. Further, 
m order to make a game administratively manageable, it may be 
necessary to limit the degrees of freedom one has with respect to 
innovation, which is an unfortunate constraint. Finally, he points 
, " out that/ it is not always clear exactly what students are learning 
m a simulation situation. ."There is no, doubt that the simulation 
technique is a powerful teaching device, and therefore is poten- 
tially dangerous unless we are relatively sure of what is beinq 
taught . " . ^ 

One wonders, in connection with the last point, if definition 
of what should be taught and learned can really be expected prior 
to development of an adequate prescriptive theory of management 
decision making. Moreover, it seems clear that so long as decisions 
are evaluated m terms of effectiveness rather than in terms of 
logical soundness, the answer to the question of whether any train- 
ing program is teaching individuals to make optimal decisions will 
-emain a matter of conjecture. /apropos the point of .how to insure 
t,hat simulations have some realism, Freedy, May, Weisbrod, and 
i^eltman (19^4) have proposed a technique for generating decision- 
task scenarios that -utilize expert judgments concerning state 
variables and transformations in much the same way that a Bayesian 
aggregator would make use of expert judgments of conditional pro- 
babilities. 

We would summarize our own attitude toward simulation training 
in the following way. The approach has many advantages. The stu- 
dent can be exposed to a variety of decision situations. Situa- 
tion parameters can be varied systematically, thus permitting the 
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study of their effects on decision-making performance. The conse- 
quences of incorrect decisions are not catastrophic, as they could 
be in some real-life situations of interest. The student's per- 
formance can be evaluated and immediate feedback can be provided 
to him, thus, presumably, improving his chances of learning. 

On the negative side of the ledger, there is first the diffi- 
culty of the task of deciding what aspects of a situation to simulate. 
Any simulation is a simplification, and if one wishes tc assure 
transfer of what is learned in the simulated situation to real-life 
situations, it is imperative that the simulation preserve those 
aspects of the real-life situation that are relevant to the skill 
that IS being trained. Moreover, the difficulty of assuring the 
veridicality of a simulation is likely to increase greatiy with 
the complexity of the situation that is being simulated. "Second, 
there is the problem of generality. Situations axe specific. One 
wants the student to carry away from training sessions skills which 
will be applicable in a variety of contexts. Simulation itself 
does not guarantee that that will occur. In fact, one might guess 
that there would be the danger of focusing on specific aspects of 
particular situations which could have a tendency to impair the 
learning of general principles. 

12.3 On the Idea of a General-Purpose Training System for 
Decision Makers" ~. ' 

A training system for decision m^ers that has a reasonable 
degree of general,ity is bound to be a relatively complex system. 
Moreover, given the current level of understanding of decision 
processes, it is unlikely that anyone would be able to design a 
system that would be certain to be satisfactory. The approach 
that seems to us most likely to produce a useful system is an 
explicitly evolutionary one, and one that involves potential users 
of the system iu its development from the earliest stages, what 
one needs to do is build a working system that represents one's 
best guess concerning what capabilities such a system should have, 
and then elaborate, extend, and improve the system in accordance 
with the insights that are gained through attempts to make use of it. 

The idea that many complex systems are best developed through 
an evolutionary process is not a new one. Benington (1964) has 
argued strongly for such an approach in the development of cominand- 
and-cortrol systems. Commenting on the fact that many systems be- 
come obsolete even before they are operational, he notes that "The ' 
principal cause of this situation is the fact that until recently 
the proposed users of these systems did not take many interim steps 
that would have helped them; instead, they waited for the grand 
solution, ivhen the development of these command-and -control systems 
was undertaken, it was thought that the design team could analyze 
present operations, project changes over many years, design a system 
for the far-off ruture, and then implement. Now most agree that 
this process just won't work" (p. 16). 
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SECTION XIII 
DECISION AIDS 

The recognition that — whether because of behavioral defi- 
ciencies or basic limitations^-men often do not perform optimally 
as decision makers has motivated the development of numerous de- 
cision-aiding procedures and techniques. The existence of deci- , 
sion aids has two somewhat opposing implications for the training 
of decision makers: On the one hand, insofar ,as ap aid succeeds 
in simplifying or otherwise facilitating the performance of some 
specific task, its existence may lessen the training demands vis- 
a-vis that t^sk; on the other hand, users of decision aids must 
be trained to use those aids. It does not follow from, the fact 
i that some training may be required before an. aid can be used 
effectively that the aid is therefore a failure; if a trained 
user of -an aid can make better decisions than a trained decision / 
maker who does not use that aid, then the aid may be sa^-d to be 
an effective one. ^ . ■ 

» 

Giv^ the view of* decision making as comprised of a variety 
of tasks , and processes, it seems reasonable to expect that initial 
decision-aiding techniques will be more successfully applied to 
some of these tasks than to others. The gpal should be, not to 
develop the grand aid for the decision itiake^*, but, rather, to 
develop a variety of aids to facilitate performance of the various 
tasks. Together, a .group Bf isuch aids might be thought of a's. -a- 
"ciecision support system" (Levit, Alden, Erickson, & Heaton, 1&74-; 
Meadow & Ness, 1973; Morton,. 1973), but the individual aids, and 
not the system, 'are probably the more reasonable' objectives toward 
v/hich to work initially. - & ^ 

. / , ' 

Another factor that some researchers have argued is highly 
relevant to the design of decision aids is that of individual dif- 
ferences. One group of investigators, for example, has character- 
. ized "decision styles" in terms of three dimensions with respect 
to which individuals are assumed to vary: abstract-concrete, 
logical-intuitive, active-passive (Henke, Alden, & Levit, 1972; 
Levit, Alden, EricKson, & Heaton, 1974). All possible combinations 
of the extremes of these dimensions are viewed as eight "pure 
decision styles" that ar6 representative of the l^pes of individual- 
ized approaches to decision making that decision-aiding systems 
must take into account. The point that these investigators make 
is that decision aids or decision support complexes, should be 
designed with particular userp, or user types, in mind. Systems 
designed for one type of decision style, they claim, may degrade 
the performance of a user who operates according to a different 
style. 
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Decision aids run the .gamut from <the types of heuristic 
principles discussed by Polya.(i957) to explicit paper and pencil 
procedures for working through some aspect of a decision problem, 
to interactive computer-based techniques. In this section, we 
consider only a few of the many ^ids to decision making that have 
been developed. The intent is not to provide an exhaustive review 
but a representative sampling of what has been done in this regard. 



13.1 Linear Programming / 

Linear programming is a mathematical technique for determi- 
ning a set of decision parameter values that maximizes or minimizes 
specified functions within certain linear constraints. The tech- 
nique is particularly, useful in solving such problems as resource 
allocation, production mix and .industrial cost contra. 1. it is 
best illustrated by a simple example. 

Suppose a manufacturer produces three products. We will 
designate the monthly quantities of these products as xn, x^ and 
X3. The products have different unit production costs, say, 
ac^f and ao, and different unit sale prices*, say, bi, b^, and . 

133. To kQep the illustration simple, we ignore the problem of 
inventories. Raw material limitations restlrict the number of units 
of products 1 and 3 that can be produced per month to c-. and c^, 
respectively* The total number of man-ho.urs available to the 
producer is n per month,, and it requires dy, d2, and d3 man-hours' 
to produce one unit of ^products 1," 2, and 3, respectively. The 
problem is to determine the, number of units of each'^pr'bduct that 
the manufacturer should produce per months in ordet to maximize 
His profit. . ' 

Linear programming is a technique for solving suoch problems, 
when solutions exist. The teqhnique involves expressi ng t he con-/ 
strain ts'^as a set of simultaneous linear equations, an3""^Iferr^ 
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searching within the ranges of the values of the independent 
variables that satisfy the equations for those values that op- 
timize the desired function. In the case of our example, the 
function to be optimized ^in this case, maximized) would be the 
profit function, i.e^, 

{b^-aj^)x^-f (b2'^^2^^2 ^^3'"^3^^3* 

When the problem involves only two or three decision vari- 
ah].es, a geometrical model of the situation can give the decision 
maker an intuitively meaningful representation of the significance 
of the various factors and, in paxjticular, of the sensitivity of 
the decision outcome to a less than optimal selection of values 
for the decision variables. When certain boundary conditions are<^ 
met, the set of parameter values that satisfies the linear con- 
straints within which the decision must be made is reprfisented 
by convex polygons or polyhedra (in the two- and three-variable 
cases, respectively) , and the solution to the optimization problem 
invariably is (or at least contains) one of the figure's vertices • 
The same principle holds in cases of more than three /ariables, 
but, of course, the geometrical model is no longer helpful. 

One of the limitations of linear programming is tae fact 
that it is applicable only to situations in which the decision 
s^ace has been fully represented numerically and th; outcomes of 
all of the admissible decisions are known. Another is the fact 
that it can be used only when the effects of the. individual deci- 
sion variables combine in an additive (linear) fashion. One can 
imagine real-life decision situations in which the effect of a 
change in the value of one decision variable depends in some way 
on the value of another variable. For exampre, hew much impor- 
tance or\e would attach to a difference in salary between two jobs 
might depend on whether the jobs also differed significantly in 
terms of the extent to which they placed one's life in -danger. 
As has already been noted in Section IX of this report, however, 
several investigators of decision making have argued that the 
assumption of additivity appears to be a reasonable one in many, 
if not most, real-life situations. Probably the more difficult 
requirement to satisfy in order to use linear programming tech- 
niques is that of adequately structuring the decision spaae and 
quantifying the salient variables. When the necessary conditions 
can be met, however, there can be no doubt of the effectiveness 
of the technique. 

13.2 Decision Trees and Flow Diagrams ^ 

Sometimes it i6 possible to convert an apparently complex 
set of written or verbal instructions concerning a problem-solving 
procedure into a decision tree or flow diagram. When such a con- 
version can be accomplished, it is often found that the desired 
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procedure is more easily and efficiently followed with the aid of 

^Sf^ fo.o^^^ original set of instructions (Blaines, 

1973; Raiffa, 1968; Wason, 1968; Wright, 1971). ^ ^ 

. The following distinction between decision trees and flow 
. diagrams IS made by Tr.iggs (1973) : . "A decision tree is an assembly 
of -individual paths in a- structure organized so that no path ever 
returns or proceeds to another part of the diagram. A decision 
flow diagram may, on the other hand, contain paths that return f 
to early parts 'of t^e diagram or, feed to other common elements. 
A decision flow diagram can be more operationally directive in its 
structure^ and less concerned with the explicit details of the 
decision process m a tree structure, at eVery node of the tree, 
the user of the diagram can exactly state by what set of chance • 
events and decisions one arrived ther.e. The flow diagram structure 
IS not always (Organized so that each such path can be uniquely 
specified" (p. 3). . - - 

The clarity and. efficiency gained by representing procedures 
requiring sequential-^decisionS in diagrammatic form have been 
recognized for some time. In such fields, as computer programming 
and systems analysis, graphic techniques ^have been employed in 
the teaching and conduct of specific programming, debugging, main- 
tenance, and troubleshooting tasks. Only recently, however, have 
formal attempts been made to assess the benefits- to be derived in 
an entertaining article by Davies (1970) the results of a relevant 
experiment by B. N. Lewis are discussed. The latter investigator 
presented a series of six problems -involving a tax regulation to 
each^of 60 subjects. One third of the subjects worked with the 
original (prose) statement of the regulation, a second third worked 
with a simplified (prose) statement, and the final third worked ' 
with an algorithmic (decisi9n tr^ee) form. The mean time required 
by the original prose group to solve all six problems was 23.4 
minute-s, compared, to. 11.8 minutes required by- the simplified prose 
group and 9.2 minutes required by the algorithm group. Mean errors 
\ in problem solution followed a similar pattern: 29%, 10%, and 8% 
for the respective groups. 

More recently, Blaiwes . (1973) compared trfs performance of 
decision makers who had been given instructions concerning the ) ^ 
construction and use of decision trees with that of decision 
makers whp had hot been so instructed. Only one of ±he ten subjects 
in the uninstructed group" gave evidence of using a deeision-tree 
approach to the solution of the four experimental tasks, whereas 
^ll^y °5 instructed subjects used it. Subjects using thb 
decision-tree approach initially required more time than 

o^t^fnT -f •r^^-^'J^'u^"^ ^^^^^ performance improved as they 
gained facility with the approach. Most importantly, subjects 
xn the instructed group performed at a higher level of ' 
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accuracy than subjects in the uninstructed group. Although •the 
possible effects due to practice cannot be separated from those 
due to problem difficulty because of *the .particular design used 
by Blaines, we regard the experiment as a demonstration of the 
ease with which the dec i sion--tree . approach can be taught to 
individuals who have not previously encountered it. 

A review of numerous attempts to apply decision trees and 
flow diagrams to the solution of decision problems (e.g., Baker, 
1967; Clarkson, 1963; Dutton & Starbuck, 1971; Horabin, 1972; 
Howard, Matheson, & North, 1972; Rousseau & Zamora, 1972; Tudden- 
ham, .19680 has been prepared by Triggs (1973) . He points out 
that the degree to which such aids can be useful to a decision 
maker will depend or( the nature of the problem that is faced. 
They tend to be most useful for situations that are easily struc- 
tured, perhaps by means of decomposition techniques advocated 
by Raiffa (1968). Triggs cautions againsLt the temptation "to make 
a complex problem tractable by forcing it into a conceptual repre- 
sentatfion witR which one knows how to cope," at the •expense of 
ignoring or eliminating critical aspects of the real problem. 
He also points out that' the task of imposing the type of structure 
on a decision pxcblem that is necessary if decision trees or flow 
diagrams are to* be used , to advantage, may be sufficiently time- 
consuming and expensive to assure i'ts ''impracticality in ^ome dy- 
namic situations in which the time for analysis is limite^. More- 
-over, forcing the decision maker to think about his problem in 
terms of a specific structure may inhibit his use of cognitive 
skills that he otherwise might bring to the task., Triggs concludes, 
however, that on balance these cautions do not negate the efficacy 
o£ the approach. Citing Zadeh's (1973) work, he notes that "even ' 
in systems ^hat are too complex or too ill-defined to admit of 
• precise quantitative analysis, 'fuzzy' algorithms and diagrams 
have the potential of being useful to the human decision maker" 
(p. IT). . ^' 

A lucid tutorial treatment of decision trees and their use 
is presented by Peterson, Kelly, Barclay, Hazard, and Brown (1973) 
in Chapters 2 and 3 of a Handbook for Decision Analysis. ^The 
handbook has been prepared for the express purpose of aiding the 
individual who is faced with substantive decision problems to 
apply concepts and procedures of decision theory to the solutj^on 
of thofee problems. ^ * 

13.3 Delphi, an Aid to Group Decision Making 

The decision maker of most prescriptive models of decision 
making could be an individual, a committee, a corporation, or a 
machine, inasmuch as such mode^ls are concerned with the decfision-* 
* making process and are indif fei^ent to the nature of its embodiment. 
>lpst empirical stud^tes of decision makinq, however, have focused 
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on the behavior of individuals. Relatively little attention has 
been given to the question of how decisions are, or should be, 
made by^ n-person groups. There are> of course, large liter- 
atures dealing with related topics such as the effec^ts of group 
organization and communication channels on problem solving, and 
the effects of grou|) pressures on individual behavior. 

One generalization that it seems safe to make is that the 
decisioh-making performance of groups may be :^nfluenced by a 
number of factors that are not obviously related. to decision 
quality in any straightforward way. Especially 'is this true^usd/en 
group -members are required to resolve problems about which there 
exist conflicting views. As Helmer (1967) puts it: . i . 

"Round-table discussions for such purposes have certain- 
psychological drawbacks in that the outcome is apt to be a ! 
compromise between divergent views, arrived at ail too oftei^ 
under the undue influence of certain faqtors inherent in th^ 
faqe-to-face situation. Th^se may include such things as ; 
the purely specious persuasion of others by the member withj 
the greatest supposed authority Jbr even merely the loudest J 
voice, an unwillingness to abandon publicly expressed ! 
opinions, and the bandwagon effect of majority opinion" (p.j 9) 

As one^eans of remedying these types of prqblems,. and of 
providing a rationale by which to combine "expert" opinions, the 
Delphi method., was created (Brown, 1968; Dalkey & Helmer, 1963; 
Helmer, 1967; Rescher, 1969). This technique requires each member 
of the group to write down his , independent assessment of the prob- 
lem or solution under study. The set of assessments is then 
revealed to all melnbers but without identification of which parti- 
cular assessment was made by which member. The pros and cons of 
each response are then openly debated and each member files a 
second assessment. Following n repetitions of this procedure, 
the median assessment is then adopted. ^ 

The Delphi procedure is reputed to be usable: ' 

"1) To determine what the operative values of a group are, 
what relative weight they have, what sorts of possible trade- 
offs obtain among them, and the like. 

2) To explore the sphere of value criteriology , clarifying 
by what criteria the values of a group come to be brought to 
bear upon actual c^Vs. 

3) To discover divergences of value posture within a group 
and the existence of subgroups with aberrant value strugj-tures. 
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^ 4) To serve as a tool for seeking' out areas of value 

consensus — or agreement as to actions and preferences — 
that may exist even when there are conflicts of value. 

5) To provide a tool for the third-party evaluation of . 
conflicts of jLnterest. ^ • 

6) To assess the correctness of value ascriptions to given 
groups . " . . ^ 

♦ > 

7) To assess the correctness of value judgments in the area 
of riieans-values" (Rescher r 1969 , p. 17 ) . v ' 

The use of a modified version of the Delphi technique is 
illustrated in a recent effort by O'Connor (1972) to apply* expert 
* judgment to the scaling of water quality. The problem was to 
assess the quality of water to be used (1) as a public supply, 
and {2)r for the maintenance of a fish^^and wildlife populacion . 
Eight experts made iterative judgment^ as to the parameters * to be -^x 
included, the relative importance weights to be assigned, and the 
tules for combination^ of indices. ^ Good consensus was- obtained 
with respect to sets of judgment ' parameters and Combination rules, 
but there was considerable disagreement on weightings-. O'Connor 
found, however, that this ' disagreement w^s not critical in the 
development of the final indices. 

An important feature of t^e Delphi technique is the fact that 
it provides a means for^ achieving group consensus wi'^fchout the need 
for the face-to-face discussion of issues'' which typifies most 
group probleip-solving methods. Thijs characteristic was exploited 
in the O'Connor study, where the experts were geographically widely 
separated and were never in direct cOmmunicatiori with iac^ other. ' , 

'13.4 Computer-Based Decision Aids V 

The potential* advantages to be gained 'from ^ap^lying the 
general computational capabilities of digital computers to deci- 
sion prob/Lems have been recognized for some time. Several writers 
have made very convinoih^rguments to the effect that botih men 
and computers have something to offer to the decision-makfng 
process, and that the nged is for the development of decision 
systems that assure a symMotic coupling of the capabilities of 
man and machine (Br iggs ^ Schum, 1965; Edwards, .J965b; Li^klider, 

1965;. Yn^a & Klem, 19^i;^ Ynt^ma' & Torgeirs^pn, 1961). 

<Jlt is not difficrult imagine';'i5i^ computer system be|ng used * 
to aid a decision maker ir\ the perfcsfmahce of essentially all of 
the aspects of decision making that we have considered in fore- ^-^^ 
goina sections of this 'report. Such a system might provide the'^ 
deciMon maker wit^-:^ 4^ta base of , facts or observations that ^re 
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relevant to his decision problem. It could serve as an extension"^ 
of his own memory by keeping a record of factors that he had in- ^ 
dicated he ought to "keep in mind" in making a decision. It could 
help him generate hypotheses, and to structure and present the 
decis:^n space. It could help him discover what his preferences 
are and to express them in a quantitative way. It ccld provide 

• graphical representations of the decision situation • It might 
^(assuming a^v^lid model of the decision problem) project the prob- 
able ct)nsequences of various action selections. It might serve 

• as an interface between two or more decision makers' collaborating 
on the same problem and facilitate the application of group deci- 
sion tec^hniques. It could do whatever computation was required* 
It could" prod the decision maker to consider aspects of the problem 
that he Otherwise n\ight overlook. It could suggest approaches or 
strategies that h^ve been found to be useful in similar problem 

. situations. It could make explicit to the decision ir^ker (either 
' b^^j^nference or by questioning of the decision maker himself) some 
aspects of the situation or the decision maker's thinking that 
otherwise would only be implicit. And so on. 

It IS in fact easy to imagine ways in which the computer 
could bemused as an aid for decision making that one can be seduced 
to thinking that the iinplejientation of such capabilities is a 
straightforward thing. In^some instances this is perhaps the case; 
in others, it^assuredly is not. The important point is, however, 
that comput'er^ased decision aids are being^ developed and quite 
sophisticated ones are likely to be operational in the near future^. 
No trai^ng program for decision makers can afford to ignore this 
fact. V ' 

In a preceding section of this report soitie comments were' made 
^ concerning simulation as an* approach to training. Given the avail- 
* ability of computer ; systems to decision makers, another way that 
simulation may be used to advantage is as an operational decision 
aid. In this case the effects, or probable effects, of selecting 
specific action alternatives can be explored by 'the decision^ maker 
befor^ he actually makes hi/s choice (Ferguson & Jones, 1969) . The 
^ projections or predictions of the aid will only be as good, of 
. course, as is the model, of the ;5ituation that produces them, and 
^^it is riot necessarily the case ):hat 'the use of such predictive 

aids will invariably lead -to imt>roved performance (Sidorsky & Mara, 
^.1968)-. The potential for this ^type of simulation is. great, however, 
and deserves more attention that it' has recaived to date. At the 
very least, such an aid can be \Jsed to help^d^ermdne what is 
possibly and what 13^ not, giving an accurate representation of the 
current state of affairs. The point i& illustrated by an experi- 
mental decision aid designed to monitor 'andfcontrol maTit|2ne 
traffic^i (Eimaltm, Prywes, & Gu^afemo, J.967J . The system v;a a com- 
posed oi\ a Iforsiatted^ data base J a seb^f- , "worker programs" which 
operatedUon the dap6. base, and a quoify languaqo v;hi<ch allov;ed*the ' 
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user to interact with the data base on line. Information that 
could be .extracted from th^ data base on request included" (1) 
past, present, or future locations of ships, (2) the number, tvpe, 
or names of ships m any geographic area of the North Atlantic' 
at a past,, present, or future time, and (3) how far is a ship 
from som'e particular place' and if ordered to change course when 
can It get there?" (p. 206). The system could provide information 
on sets of. ships satisfying some class description; for example. 
It could provide th6 distances of all ships of a given type, from 
a given destination, .and the time required to reach that destina- 
tion, assuming the necessary. chang6 in course. The system 'illus- 
trates a nice allocation of function between man and machine. 
The computer does the bookkeeping and arithmetic,, the man exer- 
ceses judgment and makes choices- Hopefully,, the choices that' 
the man makes will be the better because qf the bookkeepind and' K 
arithmeti-c that the machine does. , r ^ . 

Two of the more prominent problem areas for which computer- 
based decision aids have been developed dr planned are medicine 
and military -tactics. 



13.4.1 Computer-Based Aids for Medical Decision Making 



Among the first investigators to attempt to apply modern 
decision theory to medical decision making wer^ 'Ledley and Lusted 
(1959). During the subsequent fifteen years, many such applica- 
tions of decision theoretic techniques were proposed and tried; j 
and within the past ten years, several experimental computer-based 
systems have been developed for the purpose of facilitating 
various aspects of decision making in the medical context. Ap- 
plication? that have been explored include initial patient inter- 
viewing an^ symptom identification (Griest, Klein, & VanCura, 1973;' 
Whitehead & Castleman, 1974), analysis organization and presenta- 
tion of 'the results of laboratory tests (Button & Gambino, 1973), 
personality analysis (Kleinmunt.z , 1968; Lusted, 1965), storage and 
retrieval of individual-patient data (Collen, 1970; Greene, 1969), 
on-denfand provision to practitioners of clinical information 
(Siegel &' Strom, 1972), automated and computer-aided diagnosis of 
medical problems .(Cumberbatch & Heaps, 1973; Flisher, Fox, & Newman, 
1973; Fleiss, Spitler, Cohen, & Endicott, 1972; Gledhill, Mathews 
^& Mackay, 1972; Horrock^ & deDombal , 1973; Jacquez, 1972; Locwick, 
1965; Lusted, 1965; McGirr, 1969; Yeh, Betyar*, & Hon, 1972), 
management and graphical representations of data to aid research 
in pharmacology and medicinal chemistry' (Castleman, Russell, Webb, 
Hollister, Siegel, Zdonik,,& Pram, 1974), modelling'of physiological 
systems and exploration via simQ.lation of the effects of alternative 
(iourses of treatment (Seigel & Farrell, 1973), and training 
(Feurzeig,^ 1964; Feurzeig, Munter, Swets, 6 Breen, 1964). 
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The results of one recent study of computer-assisted diagnosis 
are particularly relevant to the questioji of when expert judgment 
should or should not be used in the decision process. Leaper U972, 
1975) compared two methods of computer-assisted' diagnosis o'£ dis- 
orders for which, abdominal pain wasVa pxM^ry symptom (e.g. ', appen- 
dicitis, divertaculitis, perforated ulcer). Comput^^ided Bayesian 
diagnose^ were performed using estima.tes o,f-probabii£i^s that were 
either fa) i'n£|rred from frequency data oollecte^rS 600 patients 
or (b)- produced hy^a. group of. clinicians. The diagnoses that resulted 
from, the computer-aid>ed method that used th^ clinicians' proba- 
bility estimates were 'marginally more accurate thdn those produced 
by ..unaided clinicians (82% versus 80%). The method that ma^e use • 
of probabilities inferred from incidence data, however, gave s-ig- 
nificantly more accurate resultc (91%). A secondary result of 
this study that* is of some interest is the fact 1:hat mos€^ clinicians 
insisted on retaining their own probability estimates, even when 
those estimates were greatly different from the, survey data and •* 
they had been informed of this' fact. , " ' 

. ' ' ' ■ 

V Thes^ results strongly suggest that relative frequency data 
should \^ used>as a, basis for probability' estimates in. preference 
to expert opinions, i£ such data A^re available. The principle 
should not be" applied, -of dourse, without 'dud re^ajd for such fac- " 
tors as the size and representativeness ,of th6 samples from whiph 
the relative freqt^ency data-^e obtained": As a general yule, the 
most defensible strategy in*> estimating probabilities would seem 
to be: use expert judgments only if a Jaore objective method is not 
feasible, as would be the case when estimating the probabilities 
of very low-frequency events or events that are not reasonably 
thought of as "f requentistic" in nature. 

13 4 2 Computer-Based Aids for Tactical Decision Maying 

Much has been written about the use of computer-based aids 
to facilitate decision making in the Context of tactical operations 
(^Tlden, Levit, & Henke, 1973; Baker, 1970; Bennett, Degah, & Spiegel, 
1964; Bowen- Feehrer, Nickerson, & Triggs, 1975; Bowen, Fee,^rer/ 
Nickerson, Spooner, & Triggs, 1971; Bowen, Halpin, Long, ][4ukas, 
Mullcfrkey, & "Sriggs, 1973; Freedy, Weisbrod, May, Schwarti:, & Wett- 
man, \L973; Gagliardi , ^ Hussey , Kaplan, & Matten, 1965; .Hanes & Gebhard, 
1966;\evit, Alden, & Henke, 1973; Levit, Alden, Erickson, & Heatori, 
1974; Sidorsky & Sdmoneau, 1970). The extent to which such systems 
an<Vaid:S haye i,ed to improved decision making is probably impo,ssible 
to determine. /is easy to be critical of this Work, however, 
bebaiise -progress Has certainly not bigoc, spectacular . And it may 
be thkt some of the decision-aiding efforts have been poorlV 
c6:!iceived. But tactical decision making is complicated and not 
thorouqtjiy understood. It is not surprisinq that there v;ould Ue 
some false starts before significant progress i^^made on this 
problem. Even fa^lse sta-^^s can provide usetul insights into a 
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problem, however; if jio thing. more/ tjiey should help to' clarif^y^^— - 
what the QMmensions of th^ problem are and to provide some clues 
concerning ti^e requirements for a solution* 

* - . ^ * * 

• Bowen, Nickergon, Spooner, and^- Triggs^ (12:70) have de^Lcribed 
several computer-based systems that have. been,, or, are I^eing, de- ' 
ve loped by the mil*itary services' to aid the decision-making proces 
in tactical s-ituations. Among the systems irhat were reviewed. are: 
the Army's T.actVical Operations. System ^TOS) — in particular , "^TOS- 
7th Ai:my7-and Tact;ical Fire* Direction System (TACFIRE) , the ^ir 




ling System i CIDHS.) , and the Navy|s integrated Op^tional Intel- 
ligence System (lOIS) . These systems* are intended to. impr>over 
tactical decision making by . facilitating data management, and mani- 
pulation, message routing, display generation, report preparatipn-, 
fire control, plannihg, resource allocation, and other ;:asks and ' 
foinctions that fall within the^pjarview of tactical operations. 

•There are two motivations for bringing s€ch systemb into the 
tactical situation. One is to unburdep the dicisiorf makeg of tasks 
that are just as w^ll perfprmed by machines, ^and thereby make it ' 
possible for him to davota more time to those aspeqts of decision 
.making 'that require hiftn^n judgment^ and expertise. ^The other is ^ 
to upgrade the quality arid adequacy' of the information on whjLch 
decisions are based. This involves inot only the problem of p;t;oces- 
sing and inl^egrating large amounts of information, but" also tliat. 
of packaging^ and presenting information in ways that ar^ well- ^ 
suited to the information-processing. ^capabilities of the human ' 
being who^^must make use of it. HbW effectively existing or con- 
teiflplated* systems realize these objectives ;Ls difficult to^deter- 
mine with much precision. ' ^ 

^ ' "' • ' 

1 It is not the purpose of this review to (^escribe particular- 
systems in detail. We will, however, .consider briefly two systems 
as i2,lustrative of those that have been developed, one intended 
for /Operational use, and one for {ise as a. training instrument. 

13.4.2.1 AESOP 

• • • 

An intensive program to develop an on-line. information-control 
system of" value to mili£ary decision makers ik the planning of 
tactical and s'tr^tegic resource allocabions was begun by .the Mitre 
Corporation in 1964. On completion in 1969, the prototype, cabled 
"An Evolutionary System for On-line Planniftg . (AESOP) to emphasize 
its .incremental approach to the generation of .qomputer-based^ 
management and planning assistance, made available to system usefs 
a range of techniques which could aid in such diverse activities 
as. data acquisition, aggregation, plan assessment and report 
preparation.. . ^ , ^ ' 
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The AESOP^ system -consists of twb major parts. "One of -these 
r-- is a -set of capabilities for storing, modifying, retrieving, and 
displaying dat^, and for performing various sorts of symbolic and 
arithmetic manipulations with the aid of a, flexible display- 
oriented user language, a light pen,, typewriter and. push-buttons. 
Details these aspects are covered in a \{ariety of program pub- ^^ 
lications, the most informative of which, are, Bennett, HainSs, and * 
Summers (1965) and Summers -and B^nnejtt (1967), f • 

; The second part of the system consists of a'^set of simulated 
^ strategic* and tactical military applications whi^ch provide a * ♦ 
context for exercising the capabilities mentioned above. One of 
the morev^ignit^rcrSfht of these is that of a Tactical Air Control 
Center (TA^ScT in which the resource ^ailocation tafek^ of a Fighter 
Sec tiop/Cur rent Plans Division are^ simulated. Since* this parti- 
cular application also^ served' as a »test)3ed for the formal. test and 
evaluation of AESOP principles*, it provides' tlje most ^comprehensive 
pijture of the strengths^ and 'weaknesses of the system. The ^ifemain- 
der. of our current summary will relate to this applictitibn and,, to 
the results 6f evaluation studies. More detailed* treatments of 
the simulation and evaluation can be found in Doughty (1967), 
]Doughty and* Feehrer (1969) > and Doughty, Feehrer, Bachand and 
Green (1969).^ ' X 

As simulated the AESOXP^ogram, .the basic ^ask of(. a Fighter 
Section revolv(3s about t^he allo&^tion (on reque^ by higher: head- 
quartei^) of tactical aircfraft to each of three mission c^'tegoiries : 
(1) oh-call close air support, (2) preplanned close air. supportty 
and (3) preplanned counter-a'ir ^nd interdiction. Under "normal" 
circumstances the total number of ready aircra^^ in near proximity 
* to prescribed target areas is less ""than, the jiumbar of aircraft 

requested, so' the planner is foxced to mal^e tradeoffs relating to ^ * 
such factors as sortie rate, flying time, time over target, and 
pro!^able degree of tacget destruction. The cumulative consequences 
^ of the^e tradeoffs are: (1) that some reque^s for support fail to 
^ be satisfied at' all, (2) some ret^ests fail to be satisfied on a 
timely basis., and' .(3 >. some requests, though satisfied on a timely 
basis, ^are not satisfied at the required level. " . * 

In this conte^tft, the tactical version of AESOP has two 
'interrelated- goals: (1)* the elimination of much of the labbxv^nd 
inaccuracy assojbia^ted with manual computation and display of ^ ready i 
resources, ^ortie rates, flying times and weapons* effects, and 
with th^ preparation of formal order? (Fragment^ary Orders) to 
squadrons implicated in a ^)lanned allocation, and (2) f acilitatioi;! 
of the problem- so lying activity of decision "makers, that is, of ^the 
judicious selectlph of squadr'ons, aircraft types , ^weapons categories, 
and so on. ♦ . y . 

^ For purposei^ of evaluation, tKe actual resouirc^ allocations 
, produced by planners using the AESOP system *w6re compared with 

4 
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those .produced by ^planners' using a simulated version of the stan- 
•^dard system in an-integr^ted series of tactical ex,ercises depicting 
^the military ^maneuvers of loyalists and insurgents^ during a ten- • 
day ^limited war. Experimental se9igiDns ODegan with briefings re- 
lating to orddros' of battle, political and military activity, and 
Joint Task Force requests for support of loyalist objectives. 
AESOP and Manua^ Planning teams then adjou'rned to commence allo- 
cation activities in response to the simulated JTF requests. The 
experiiTteat ended each day with the (automated or manual) produc- 
tion of squadron Fragmentary Orders • ' * 

Each planner, whether^'operating the manual or AESOP system, 
w^s required to generate an alloca^tion which repres%±ed, in his 
judgment, .the best tradeoff among four criter^ia (listed in de- 



creasing order of importance) : 

1. Sa'fed^sfaction of requested level of damage 

^^atisfactiofi of requested time over target 

Minimization of- use of- recycled aircraft 
(i.e.. , of sortie rate) . , ^ 

I, - / 
Minimizatioiff of (total) flying time. ' • 



\ 
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' The results or the evaluation study contained few surprises. 
In those , aspects of planning activity for which -AESOP provided 

. direct assistancv,e, perforrmance of those usiri!^ the system was su- 
perior. In ;those aspects, for which assistance was not provided, 
planners 'in the 'two systems performed at approximately equal levels. 
The net performance of AESOP planners was superior to^that of 

'manual planners v/it'h, respect to plSn quality am^d production ef- 
ficiency, 'a findincj that must be assessed in light of the fact 
that the larger portion of "the task was fairly routine and rtequir^d 
little creative ab^pLlity. 

I^ is important to notg that thfe AESOP system provided- -fto 
formal prog^dural aic^s to the decision>maker such as decision al- 
go/:ithm^, Ifnear programming solutions, etc. .What benefits accrued 
to users of the .^^tem during the more creative phases of their 
^task seamed to r^esult from a combination of^. indirect factors. jEt 
appeared to be the case, for example/ that planners could more 
easily comprehend the* extent to.whi'ch resources' would be "strained" 
and, ther'&by, develop a better "feel" for the nominal foi-m of their 
plan prior to its production. This afgpreciation for the difficulty , 
,of the problem with which they* were faced or\ a particular day was 
materially aided by the concise nature of the^displays provided 
\ by the^. sys^eft. Planners who used tllie system were in a mucAi better 
'^position to monitor tl"^eir own progress while solving the problem 
"than were those whdse appreciation of the demands of the situation 
had to b6 assembj.ed from groups of formal do$:uments. 
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It appeared also to be the case that,, since the system/per- 
-formed routine aspects *autoin^1^6ally, more time was "ava^ai^le for 
creative- activities and for reSrteratiQn of plans, 'On sevei^al 
occasions AESOP planners attempted successfully to pro,duce series 
of .allocations of progressively greatet merit.and stopped only 
when they were totally sat^isfied with their efforts.* 

3(3.4.2^2 TACTRAIN ' . ' 

* 

•Thfe. Tactical Train ing GTACTIlAl^) -facility was developed Jby 
the Electrxc Bbat Division of General Dynamics, partly as a**^ \ 
demonstration that a modest computer with a CRT display could 
be employed -in the- training 'of decision-making' skills and partly 
a^^ an experimental tool for eval*uation of alternative tactical 
display/interrogation formats. Details regarding computer and 
display equipment, software and tactical prqblem parameters used 
in the system and a ^specific- configuration -employed lar^ ^st'em. 
evaluation are discussed by Si^drsky and Simoneau (1970) • The 
summary^ presentation below draws heavily on their discussion • 

The TACTRAIN system provides an opportunity for- the dectisibn 
maker £o take on the role? of. a commanding officer of a submarine 
on an ASW search-a^^id-destroy mission. His specific .task is to 
maneuvei^ in such a v;ay that, he sim\iltaneously, maximizes the prob- 
ability of destroying a simulated enemy ship and minimizes the 
probability thailf the enemy ship will destroy him. Herishooses a 
maneuver by selec^ting a speed," a* depth, a firing/ range/ and a 
quantity of torpedoes, each from among ^five aiternati^^es. The 
choices are constrained to be consistent with th6 operating 
characteristics of own and enemy ships, the parameters of -ow^' ^ 
ship's weapons complement, and specific sourfa* channel, topo-* 
graphic and bathythermal conditions. The maneuver implied by 
the alternatives that are chosen is then evaluated with respect, 
to each Of four criteria: (1) the probability that own ship would 
be able to detect ^the enemy ship, (2) the ^Jrobability that the 
enemy ship would be able, to detect own ship, (3) the probability 
that own ship would be -able to'^destroy the enejny ship, gi^^n the 
maneuver and weapon characteristics, and (4) ' the probability that 
the enemy^'Ship, would he' able to destroy own ship/ 

^ While solving .particular tactical problem, the officer ^ 
can retrieve information ^^ored in the system ^by interrogating 
i;he display with a light p'en, , Appropriate interrogations lead 
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*A qu^si-lin^ar program to aid this strategy at a. formal level 
wis later developed hy Feehrer (1968) f^r the AESOP (PAQC planning 
activity. J , ^ - t • " 
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to one of two categories ofS^displaj^ (1) prior to d "command de- 
/" graphic displays of the "tactic^ 



ca^sion," graphic displays of the "tagtical effectiveness" asso- ✓ 
ciated with thB. choice of 'a particular alternative on each tac- 
tical dimension (speed, range, etc.) with, respect to ^ach of the 
• four cfitep^-^availAbie prior to a cqjniftand decision, and (2) , ^ 
alphanum^;3.c displays^ revealing the, outcome, of the, maneuver ; the • 
number of (quality) poii\ts to be .assigned to the outcome, ^ a^9d 
the buijiulatiA^ number of points acquired as of the end of' the * \ 
experimental' trial ii^- question — avaiiai)le following a command 
decision-. . - • ' ' 

.The tdevelopers of TACTRAIN see it making at least* two valuable 
inputs .to the learning processJ:of *the decision maker. Firsts it 
^rovid^s immediate* knowledge oi the conse^uenp^s -of a decision. ' ' 
The decision maker discovers ver,y quickly whether ^he destroyed 

^ the enemy ship and whether hi^ own ship was ^destroyed in the ^ . 

/process^ ^Jorepver, he is provided with an a^thmetic ..leasure, 
howe^/e^r arbitrarily derived, of his pumulati^e 'performance. 4 

• ' Second,^ the deci'sion-jtnaker is provided, via ith^^ ^^spl^y, 
with^ a graphi^^b^tr^yal of the interactions between tactical and 
environmental'-^ri^ble^ and their relationship to tactical ef- 
fectiveness ^s represenjisd *b^detectioQ/coujiter (detection and 
hit/miss outcomes. Mid, inasmuch as^thk^tactical problem unfold^ 
, over time, the decision maker also gains an appreciation for the 
changing"»complexities of* "these' interactions and for the need 
for timeliness in his* decision.' ' \ , , 

1>.4.3 Computer-Based Decision Aids and Training ' • 

It seems highly 'pxobab']^ that many attempts to develop com-, 
puter-based decision aids' wil'l fail in th^e sense that £he aids^ 
that are produced will not measure up to the expectations of • 
theiir dgveloperg^. This is ^not necessarily *£ailure in a larger 
view, however, if Idhese attempts -l^ad'yto a better understanding 
,of the decision-making procfess — as one.Aight reasonably llbpe that ♦ 
they will. To the extent tbat .these effbrts->(ao le^d to new 
insights into various aspects of the .decision -making process,* they 
will -have direct impac^ on, training curricula. 

._^,the extent tha't Specif ic systems prove to be effective ^Z' ' 
aids m' operational situations., they will constitute new to. Is V' 
5^ith which decision makers will have i.o work. Thfus, their, 
existence will represent a new training need,* namely the need 
to train the users of ^th6se aids. 

Perhaps the most challenging way in which the development of 
increasingly Sophisticated computer-based systems relates to 
training is in the potential that' these 'systems represen^t for 
providing training for their users. Critics of the 'idea of 
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computer^assisted irvstruction can correctly point out that'tfie 
results of endeavors in this area have not measured tm to #ie y 
• expectations that were fostered by many of the early Whusiasts' 
for this use of computers. Very real progress in the area -is ' 
.being made, however, and it may pro^e to be the case that th6 ' 
e^rly enthusiasts erred" on}.y in failing to appreciate the • - 
difficulty of some of the problems. that hid to be so^Ived and t^ie 
^^'il^a.J^f ^ ^® required, to solve- them. There is no ^u^stion 
but that. computer systems that are intended to be used by people 
interactively on. complex problem-solving tasks can ,be given- the ' 
capability to provide much/^ of the. training.- that "li required, 
both to i?iitiate users and ,to' bring users from' nebf)hyte to expert 
status. The potential ,gains to be": realized, by building such ' / 

training capajail it ies into operational systems 'suggest that this / 
possibility IS worth far jnore attention than it has yet received, I 
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